upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Usability Evaluation of Email Applications by Blind Users

Brian Wentz and Jonathan Lazar

Journal of Usability Studies, Volume 6, Issue 2, February 2011, pp. 75 - 89

Article Contents


Research Methodology

The following sections discuss the application selection, the pilot study and modifications, participant selection, and data collection and configuration.

Application Selection

We selected Microsoft Outlook 2007, Outlook Express, and Mozilla Thunderbird/Sunbird as the desktop email applications for this usability study, and Outlook Web Access 2007 Light, Gmail, Yahoo Mail Classic, and Hotmail for the web-based email applications. We chose Microsoft Outlook Express and Outlook 2007 due to the high usage of the Microsoft products during the web-based survey that we conducted in 2009 (Wentz, Hochheiser, & Lazar, 2010). Mozilla Thunderbird/Sunbird was included due to the growing popularity of other Mozilla products. Outlook Web Access was included in this study due to its use in many businesses. Also, when we contacted Microsoft Corporation about the accessibility of its email products, they told us that it was their preferred version for users who are blind, have low vision, or require screen magnification (Microsoft Corporation, 2007). We selected Gmail, Hotmail, and Yahoo Mail Classic due to their high-ranking popularity during our previous web-based survey of blind users (Wentz, Hochheiser, & Lazar, 2010).

Pilot Study and Modifications

Based on the feedback and testing process of two pilot study participants (both of whom were blind), we made some changes to the testing process and questions. For example, a different keyboard was used for the actual testing based on what we were told was a preferred layout of the Home, End, and Delete keys. During a pilot test of Gmail, the ARIA-enhanced version of the Gmail calendar was also tested as a possibility to use for this study, but because no major differences were discovered and because ARIA is not supported by all web browser and screen reader versions, we decided that the standard Gmail calendar would be used. A task to ask the participants to determine the total size of the email account was removed from the task list after the pilot studies, because more than half of the email programs being tested did not have this information available. Our pilot study concluded that there would only be enough time to test two email applications per user, and that the average testing session would take 3-4 hours.

Participant Selection

When we recruited participants for this study, we asked that they be completely blind, screen reader users not able to use screen magnification, and at least 18 years of age. All participants were current email users, and had at some point used email for work-related purposes. We sent recruitment emails to members of both the Maryland and Pennsylvania chapters of the National Federation of the Blind in order to obtain participants, and some members of those chapters also advertised the project to other groups and mailing lists. Fifteen participants were involved in the usability testing (not including the two blind participants in the pilot). In difficult to reach user groups such as those with disabilities, it is considered to be valid to use self-selected sampling methods (Lazar, Feng, & Hochheiser, 2010). Also, because there is no central directory of all blind individuals, a true random sampling would be technically impossible. The goal of this evaluation was to conduct exploratory research with a group of blind users, regardless of their statistical representation of the larger population of blind users.

Data Collection and Configuration

The actual testing was conducted from January through March 2010. The testing was conducted by using an Acer Aspire One netbook (Intel Atom 1.6Ghz processor and 1GB RAM) with the Windows XP operating system and Internet Explorer 8. An external keyboard, external speakers, Verizon Wireless broadband (for testing the web-based applications) and JAWS 10 (screen reader software) were used for the usability testing. Data logging software (REFOG Keylogger) was used to record the keystrokes typed, and a stopwatch was used to record the time spent on each task. We spent time creating test email accounts that could be used for each email application but would also make sure that no personal data of the participants was used. Each email application was pre-loaded with fake email messages, calendar appointments, and contacts (consistent across email applications).

We started each testing session by asking each participant questions about their background and email experience, and then we tested one desktop and one web-based email application with each person. We read each task to the participants, and then they tried to complete each task without our assistance. We used short, descriptive scenarios to accompany each task (such as “You just got back from a lunch break and want to read the most recent email message in your email program”).  Each task was timed and recorded for completion, and participants could choose to stop the task at any time. We noted problems and comments when they were mentioned by participants or observed by us. Each application was tested between three to five times. After each application was tested, participants summarized what they liked or disliked about the application. For the actual study, usability testing took an average of 3.3 hours per person. Table 1 lists the usability tasks that were used for all the desktop and web-based email applications.

Table 1. Tasks Used for Email Usability Testing

Table 1

Previous | Next