upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Usability Evaluation of Email Applications by Blind Users

Brian Wentz and Jonathan Lazar

Journal of Usability Studies, Volume 6, Issue 2, February 2011, pp. 75 - 89

Article Contents

Web-Based Email Applications

Gmail, Hotmail/Windows Live, Outlook Web Access 2007 Light, and Yahoo Mail Classic were the web-based email applications that we evaluated. Because there were four applications and 15 users who each tested one web-based application, all web-based applications were tested by four users, with the exception of Hotmail, which was tested by three users. Words such as “frustrating,” “cluttered,” and “difficult” were comments from many of the users when using the web-based email applications.


Focus of a screen reader refers to the position of the cursor where the screen reader is reading the text on the screen. A common problem that participants experienced with Gmail and other web-based email applications happened when they were using the arrow keys or tabbing through items on a screen, and the focus could be lost to the web browser menus and URL field, which caused confusion for several users.

Difficulty finding the “send” and “delete” links was experienced by two users. The calendar and contacts also created navigational problems for many of the users. Only one user was able to create a calendar appointment, and no users were able to successfully delete a calendar appointment. This was primarily due to the problems they experienced when they tried to navigate (with a keyboard and screen reader) through the application. Using the contacts also presented a problem for most of the users, and no users were able to successfully add a contact. One of the problems that created confusion for participants was the “Add contact” link on the left side of the Gmail interface. This is not really a means of adding a contact to the contacts list but rather a link to send Gmail chat invitations (see Figure 2). Overall, three users noted that the Gmail interface was difficult to navigate. Due to the popularity of the application, many of the users had previous experience with Gmail and noted that the basic HTML version of Gmail is easier to navigate with a screen reader; however, it is missing features that the standard version includes, such as the ability to manage contacts and check spelling (Google, 2010).

Figure 2

Figure 2. “Add contact” confusion in Gmail: A sighted user can visually see that the “Add contact” link is related to the “Chat” feature, but when a blind user is listening to links through a screen reader, it seems to be a method of creating a new contact.

Findings Specific to Blind Users

A sighted user restricted to using a keyboard could also lose focus in the web browser menus, but at least it is obvious from a visual perspective what has happened, and the user can navigate forward or backward to refocus the cursor on the target screen area. A sighted user may notice that the problematic “Add contact” link is grouped within the “Chat” area of Gmail, but this is not obvious when listening to the links while using a screen reader. The limitations of the basic HTML version of Gmail would affect any user who chooses to use this version; however, because basic HTML is often more usable for screen reader users, they are essentially forced to choose between usability and less functionality.

Hotmail (Windows Live)

Hotmail (now called Windows Live) repeatedly exhibited the problem of users losing focus in the menus of the web browser. The high number of links on the page that were not directly related to the application navigation took users a long time to navigate through. A significant login problem was noticed by users, in that once a user has logged into Windows Live at least once and later returns to the application, the login area changes to a large, JavaScript sign-in button that becomes particularly inaccessible through screen reader and keyboard navigation (see Figure 3).

Figure 3

Figure 3. Windows Live sign-in button problem: This large JavaScript button would be more accessible to screen readers if it were created with CSS.

Findings Specific to Blind Users

The number of links in the Windows Live interface is a problem that would affect any user who is restricted to keyboard navigation of the interface. Because JavaScript navigation can often be inaccessible, the design of this button could simply be changed to CSS creating the same visual effect but also becoming accessible to screen readers.

Outlook Web Access 2007 (Light)

Outlook Web Access 2007 Light demonstrated the highest completion rate and the most favorable completion time of all the web-based email applications that we tested. While many aspects of this application were usable, the contacts and calendar did have some problems. When adding a new contact, users lost focus in the browser menu, because the only “Save and Close” button appears at the beginning of the tab order and at the top of the screen (see Figure 4). Users also had difficulty when navigating the calendar, and they often lost focus in the web browser menus.

Figure 4

Figure 4. “Save and Close” button location problem in OWA 2007 Light: The tab order starts with “First name” and never takes the user back to “Save and Close.” A blind user will not know that this button exists back at the top of the window, because using the tab button never takes the user back to this button.

One other major issue that should be noted is that while the light version of Outlook Web Access 2007 is the recommended accessible version, the actual selection of the light version itself presents a usability problem when using Internet Explorer. The tab order on the login page of Outlook Web Access includes a checkbox that can be selected to use the Outlook Web Access Light, but it is above the username, password, and “Log On” button so that a blind user would not realize that the option to select this was present unless the user moved backwards through the tab order (see Figure 5). The Outlook Web Access interface does not give users the choice of using the standard or light version but rather forces all users to use the light version when used in other browsers other than Internet Explorer, such as Google Chrome, Mozilla Firefox, or Apple’s Safari. Overall, users did note that the Outlook Web Access 2007 Light interface was more usable than many web-based email applications, and Table 3 showed it to be more usable than all but one desktop application (Outlook Express) when comparing the percentage of completed tasks.

Figure 5

Figure 5. Illogical positioning of the light version selection in OWA 2007 Light: The “Use Outlook Web Access Light” checkbox should be before the “Log On” button because otherwise it is only obvious to sighted or experienced users that this option is there.

Findings Specific to Blind Users

The problem with the tab order positioning of the “Save and Close” button in the Contacts section is a usability problem that would affect any individual using a keyboard alone for navigation; however, because it is only visually obvious that this button exists, it would be even more unusable for blind individuals. This is similar to the problem with the positioning of the “Use Outlook Web Access Light” checkbox on the login screen for Outlook Web Access.

Yahoo Mail Classic

Yahoo Mail Classic was not only the lowest ranking web-based application, but it was also the lowest ranking email application that we tested during this study. Sixteen out of the 17 tasks had completion rates of 50% or lower, and three tasks were not able to be completed by any of the users. As with other web-based email applications, users lost focus in the web browser menus during many of the attempted tasks. The drop-down menu that appears when “New” is selected to compose a new email message was difficult for some of the users to navigate. A script was used to create new folders, and because web browser security settings may block scripted windows, the warning that the script was blocked was not apparent to users, and this resulted in no user being able to complete the task of creating a new folder.

The contacts and calendar presented usability problems for most users, with the “Add Contact” button not working for one user and others being unable to simply navigate the contacts interface. The calendar was also noted to be difficult to navigate. For Yahoo Mail Classic, in general, users complained about the high volume of advertisement content and that the interface seemed to be cluttered and difficult to navigate. Users who had previous experience with Yahoo Mail noted that the no longer available basic HTML version was easier to navigate but was missing some of the features contained in the standard version. One final problem experienced by two users was the “All-New Mail” link. Our users assumed this link would check for all new email, but in reality this is a link to take the user to a newer interface for Yahoo Mail (refer to Figure 6).

Figure 6

Figure 6. Confusing “All-New Mail” link in Yahoo Mail Classic: When listening to links through a screen reader, it sounds to the user that this is a way to read all the emails that have newly arrived.

Findings Specific to Blind Users

The problem of lost focus was discussed earlier as disproportionately affecting blind users, and the drop-down menu for composing a new email is something that is only obvious to a sighted user. The blocked script to create new folders is a problem that would affect any individual whose browser is blocking scripted window. The problems in navigating the contacts in Yahoo Mail Classic were related to confusion when navigating the interface with a keyboard and screen reader. A sighted user restricted to keyboard use might experience navigational problems, but these could most likely be resolved through the benefit of seeing the interface. As such, this is a problem that is more accentuated for blind users. The high amount of advertising would be frustrating to all users; however, non-essential content and links are even more frustrating if you are forced to audibly listen to at least parts of the advertising in order to navigate the interface. A sighted user has the option of quickly visually scanning the interface to locate the areas of relevance. The confusing “All-New Mail” link might be confusing to all users, but when quickly read through a screen reader the confusion is more evident.

Previous | Next