upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Usability Evaluation of Workplace-Related Tasks on a Multi-Touch Tablet Computer by Adults with Down Syndrome

Libby Kumin, Jonathan Lazar, Jinjuan Heidi Feng, Brian Wentz, and Nnanna Ekedebe

Journal of Usability Studies, Volume 7, Issue 4, August 2012, pp. 118 - 142

Article Contents


Results

The following sections discuss the results for the social networking, email, calendar, price comparison, and text entry/note taking tasks.

Social Networking

All 10 participants were able to complete the Facebook tasks. Some users required no assistance, and some users required encouragement or hints during one or more tasks. Seven out of 10 participants rated this category of tasks as “easy” or “very easy.” The time each participant spent on five tasks and the ratings are listed in Table 5. The results for each task will be explained more fully below in the paragraph for each task.

Table 5. Time (in Seconds) and Participant Ratings for Social Networking Tasks

Table 5

1: Login to Facebook

Participants took an average of approximately 3 minutes to login to their Facebook account. Most of the time was spent on entering the email address and password. Most participants knew how to use the basic keyboard, but many of them did not know how to switch to (or did not feel comfortable with) the uppercase and number mode on the virtual keyboard. For example, P7 kept entering the wrong password and eventually the researchers discovered that the user entered all letters that should be capitalized in lowercase. P7 did not realize that error until it was suggested by the researchers that P7 make sure that that password was entered with the proper uppercase and lowercase (note that, because the passwords are for the participant’s own personal account, and because the password is often shown on the screen as ********, it was often not possible for the researchers to determine why a password was not entered properly). Because of the password problem, P7 took almost 6 minutes to login to Facebook. Most participants noted that they saved their email and password in their home computer, so logging in with a password was typically not an issue, because the passwords were automatically entered when using their home computer. This also means that the participants often did not have repeated practice with entering their password every time that they logged in.

Task 2: Search for a Facebook page

Participants took an average of less than 2 minutes to search and locate the National Down Syndrome Congress (NDSC) page. Some participants spent longer time on this task than others primarily because they typed very slowly, such as P9. Figure 2 shows the search area on the Facebook home page with auto-suggested results.

Figure 2

Figure 2. Screenshot of the auto-suggested search results on the Facebook interface

Task 3: Send a message

The send message task took a longer time primarily because participants needed to type in (compose) and send the message. An average of 5 minutes was spent on this task, but there was a large variation in time among participants. The variability appeared to be partially related to familiarity with the iPad. Criteria for the study were that users have familiarity with touch-screen use, but they did not necessarily have familiarity with the iPad. Fast participants (such as P4) only spent less than 2 minutes to type and send the message. Some participants (such as P2 and P9) spent 9 minutes or longer on this task alone. Some participants were not familiar with the interface for writing a message on Facebook and could not figure out how to compose a new message. An illustration of this is P2, who sent a message (rather than by composing a new message) by replying to an existing message. Behaviors and personality characteristics, as well as learning strategies, also affected the time spent. P9 spent most of the time (nearly 10 minutes) trying to decide who to send the message to, despite receiving clear instructions and encouragement from us that any Facebook friend could be chosen. Some participants asked questions if they were having difficulties, and others used trial and error.

Throughout the entire data collection process, participants demonstrated various different typing styles. Most participants used multiple fingers on both hands, but in different ways. Some participants were able to use multiple fingers on both hands to type. For example, P2 used both second and third fingers on both the left and right, as well as a fifth finger infrequently; P5 used the left and right second fingers together, and infrequently used the third fingers; and P4 and P6 used all fingers on both hands from the start. Some participants (such as P3) started out only using the left index finger, then, as time went on, increased their use of other fingers. Some participants seemed to have different roles for their fingers (P7 typed using the left and right second fingers, but used the third fingers only for scrolling and the thumbs only for hitting the spacebar). One participant (P9) only used the right hand on the iPad, and at no point used the left hand. More detailed discussion on typing patterns is presented in the section on note taking (see the Text Entry/Note Taking section).

Task 4: Identify a friend’s status

Participants took an average of less than 2 minutes to identify the status of a friend. The majority of the participants completed this task very easily. P9 again spent a long time on this task because of difficulty deciding which friend to pick.

Task 5: Logout

Logging out of Facebook took an average of 40 seconds. The majority of the participants directly went to the right link and logged out. Two participants clicked the “X”’ (Exit) button on the right corner of the window to close the browser.

Email

A majority of the 10 participants were able to complete the email tasks. Participants used their personal Web-based email accounts for these tasks because it would have been confusing to users with Down syndrome to request them to use an email account that was not actually theirs. Because the email tasks have some similarity across email applications, the findings related to email usage do have value. Participants experienced more problems with email tasks than with Facebook tasks. One problem that was encountered was that the Gmail and Yahoo app on iPad do not provide a method to add a contact and then use that contact’s email address to address a new message. Three participants were technically not able to complete the task related to address book/contact list because of the lack of this functionality on Gmail and Yahoo. Six out of 10 participants rated this category of tasks as “easy” or “very easy.” The time each participant spent on the four tasks and the ratings are listed in Table 6. Note that in some cases, Web sites were used, and in other cases, participants were automatically re-routed from the Web sites to the iPad versions, which is out of the control of the researchers and the participants.

Table 6. Time (in Seconds) and Participant Ratings for Email Tasks

Table 6

Task 1: Login to email application

Most participants used the same email and password for login to both their Facebook and email account. This task was very similar to the login task of the Facebook account. Participants took an average of approximately 2 minutes to complete this task. This was, on average, a minute shorter than their first log in to Facebook.

Task 2: Create a message

Participants took an average of 4 minutes to create (compose) and send the message. Some participants had a hard time finding the “create message” icon on the screen. The create message icon on the Gmail app was difficult to understand and associate with the create/compose action. Figure 3 shows a screenshot of the icon on the Gmail app that was not intuitive to find for some users. The Yahoo! Mail interface used a similar icon for the create/compose action on its tablet version of Yahoo! Mail interface. One participant (P2) refused to complete this task because the user only read emails and had never sent or responded to any emails. The user’s inbox contained 695 messages, with no replies. This is an interesting case because this information only flowed one way, and two-way communication had never been established. This participant skipped the remaining two email tasks. In general, the participants preferred to respond to existing emails, rather than create new email messages.

Figure 3

Figure 3. Screenshot of the create/compose link on the Gmail iPad interface

Task 3: Add contact to address book/contacts

In addition to P2, three other participants were not able to complete the task related to the address book/contact list because the Gmail and Yahoo! app on iPad does not provide this functionality. For the six participants who completed this task, an average of 3 minutes was spent by them in order to complete this task. Some participants had difficulty locating the address book/contact list icon.

Task 4: Send email to contact in address book/contacts

Six participants completed this task and took an average of approximately 3 minutes on this task. It was most likely completed easily because the participants had just entered the contact information in the previous task and were able to click it right away to start the message.

Calendar

Four participants (P1, P4, P6, and P9) had previous experience using electronic calendars. Participants were asked to locate and create an event on the iPad calendar app. Most participants easily located an existing event, but had more difficulty creating the event. Six out of 10 participants rated this category of task as “easy” or “very easy.”

Table 7

Task 1: Locate an event

Nine out of 10 participants were able to locate the event on the calendar in less than 1 minute. P8 spent 2 minutes on this task because the participant spent some time first in a different calendar month. After a while, P8 was able to switch to the year view, select the correct month, and finally the correct day. Some participants used the Calendar app’s search feature to locate the event.

Task 2: Create an event

Creating an event seemed to be challenging for most of the participants. Only the first participant completed this task easily in about a minute. The other nine participants spent an average of approximately 6 minutes on this task. Some participants had problems locating the “add an event” icon (a plus sign located in the lower right hand corner of the screen). After they found it, the sensitive touch-screen interface sometimes removed/closed the box from the screen. For this task, the participants needed to specify the start time, end time, and the location of the event. Some participants had problems with the scrolling gesture in order to select the appropriate time. Figure 4 shows the time selection interface that required the scrolling gesture. Some participants accidentally tapped off the dialog box, which closed the box, forcing them to reopen, and restart the task. This happened multiple times for some participants, and it took them a substantial amount of time to figure out how to do it right.

Wishart (1993) commented on the use of social skills and charm to avoid doing tasks by people with Down syndrome. P10 took 540 seconds to create the event and was a prime example of using charm and social skills to avoid a task. The task involved creating a birthday party event on the calendar, and it took a long time, not because of difficulty with the interface, but because the participant was negotiating for a longer party. P10 did not think from noon-2 p.m. was long enough for a birthday party and refused to perform the task, despite our indication that this was a fictitious birthday party. The participant would only perform the task once we agreed to a longer time for the party.

Figure 4

Figure 4. Screenshot of the time selection interface requiring the scrolling gesture (also, note the plus sign, very small, in the lower right hand of the iPad screen)

Price Comparison

All participants were able to complete the price comparison tasks successfully and within a reasonable amount of time (see Table 8 for time performance for the price comparison tasks). Six out of 10 participants rated these tasks as “easy” or “very easy.”

Table 8. Time (in Seconds) and Participant Ratings for Price Comparison Tasks

Table 8

Tasks 1 and 2: Find and compare a book price from two different Web sites

Participants spent an average of more than 4 minutes to find the book on the Barnes and Noble Web site (Task 1) and less than 2 minutes on the Amazon Web site (Task 2). All participants spent less time on the Amazon Web site than the Barnes and Noble Web site, which may be partly due to the longer URL of the Barnes and Noble site. Some participants made errors when entering the URL of the Barnes and Noble Web site, and some participants used Google search to locate the Barnes and Noble Web site rather than typing the URL. It also could be related to familiarity with the task, as there could have been a learning effect. At the Barnes and Noble Web site, participants potentially could have learned how to successfully complete the task. At the second Web site, the Amazon.com site, the participants could have carried over and generalized their learning of the price comparison task. They were also familiar with the title of the book that they were searching for by the time they were typing in that title at the Amazon.com site. Figure 5 shows a screenshot of the Google search feature built into the Safari Web browser that was used by some participants to find the URL of the Web sites.

Figure 5

Figure 5. Screenshot of the auto-suggest Google search feature built into the Safari app

Some participants were able to take advantage of the “auto-suggest” function in the iPad search box, which saved them some time when using Google. Auto-suggest worked not only on general searching for the sites, but sometimes worked on the search engines within the product Web sites. For instance, Figure 6 shows the auto-suggest function working on the Amazon Web site (auto-suggest worked more consistently on the Amazon and Barnes and Noble sites than on the Staples and OfficeDepot sites, as discussed in later sections). Other participants did not take advantage of that function and typed in the entire name of the book with authors’ names. Once the search results appeared on the screen, some participants had problems identifying the specific book and took a substantial amount of time to find the specific book.

Figure 6

Figure 6. Screenshot of the auto-suggest search feature on Amazon.com

Tasks 3 and 4: Find and compare a digital recorder price from two different Web sites

Participants spent an average of 204.2 seconds (over 3 minutes) to find the digital recorder on the OfficeDepot Web site and 176.4 seconds (just under 3 minutes) on the Staples Web site. One participant wasn’t able to locate the search box on both Web sites and browsed the menu to look for the recorder, which took much longer time than searching. That participant was not able to find the item on the OfficeDepot Web site. Auto-suggest did not consistently work on the two Web sites used for the digital recorder price task, so that was an unpredictable factor. Note that the fourth task (OfficeDepot) took, on average, more time than the third task (Staples), possibly meaning that there were no learning effects, or that the Staples site was easier to use than the OfficeDepot site, or that the unpredictable appearance of auto-suggest played a role. Regardless of which site, which task, or whether auto-suggest worked on the site, there was generally a high level of task success for the price comparison tasks.

Text Entry/Note Taking

Participants spent an average of just under 8 minutes (473.8 seconds) to enter the text using the touch-screen virtual keyboard. All participants had the text printed on a handout. Figure 7 shows a screenshot of the iPad virtual keyboard. The text entry task was ranked as the easiest among the five categories, with nine out of 10 participants rating this task as “easy” or “very easy.”

Figure 7

Figure 7. Screenshot of the iPad virtual keyboard

Performance statistics

To compile the performance statistics, four researchers independently counted the total number of words entered by the participants. Regarding error coding, one researcher examined the text first and proposed a code list with eight error categories. Then four researchers independently coded the errors in the text entered by each participant. The word count and error coding of all researchers were compared by one researcher, who identified all inconsistent codes, re-checked the original text, and fixed incorrect coding items. The final coding list contains nine error categories, which are listed in Table 10.

Table 9 lists the typing time, total number of words entered, speed (words per minute), total number of errors, error rate, and participant ratings for text entry tasks. The text entered by P6 was not saved due to technical problems. So the statistics are based on nine participants. Participants entered 10.4 words per minute on average. There was a large variance in the text entry speed among the 10 participants. The slowest participant (P9) only entered 3.5 words per minute on average, while the fastest participant (P4) entered 17.7 words per minute on average. The error rates among the participants also varied substantially, ranging from 1.5% to 36.5%.

Table 9. Time (in Seconds), Total Number of Words Entered, Words per Minute, Total Number of Errors, Error Rate, and Participant Ratings for Text Entry Tasks

Table 9

Nine types of errors were identified (see Table 10). The most common errors were incorrect capitalization (32.6%), followed by missing punctuation (26.1%), misspelled of words (12.0%), missing space (7.6%). Missing word, extra word, and extra punctuation each contributed to 5.4% of the total number of errors. The least common errors were extra space (3.3%) and incorrect punctuation (2.2%).

Table 10. Types of Errors and the Percentage of Occurrence

Table 10

Although the error rates of the participants seem to be high, the majority of the errors were minor ones that were unlikely to affect comprehension of the text, such as incorrect capitalization and missing punctuation, which together account for approximately 60% of the errors. Some of the errors might even be considered common among neurotypical users, especially when emailing or text messaging, such as using the @ sign instead of the word “at” and not using appropriate capitalization.

Some participants were able to enter the text with both high speed and high accuracy. For example, P3 entered the text fast (15.8 words per minute) with only one error. The punctuations were all correct with the only exception of the colon. All letters were capitalized as needed with appropriate spacing. The text entered by P3 is shown below:

The weather will get much colder, so I need to remember to wear my hat and gloves. I'm going to meet my friend at 4;30 p.m. to go to the new movie "War Horse." Then I need to check the bus schedule to see if there is a different schedule on Mother's Day, which is Sunday, May 13th.

P2 missed all periods in the text yet entered all other punctuations accurately, including the colon and apostrophes (see below).

The weather will get much colder, so I need to remember to wear my hat and gloves I'm going to meet my friend at 4:30pm to go to the new movie war horse then I need to check the bus schedule to if there is a different schedule on Lincoln's birthday, which is Sunday , February 12th.

It is clear that there is a gap between the ability of P7 and that of the remaining eight participants. P7 is the only participant who typed very slowly (3.7 words per minute) with a very high error rate (36.5%, the highest among all participants). P9 is similar to P7 in efficiency, but typed the text with only three errors, which is the second lowest among all participants.

Typing patterns

Table 11 summarizes the typing patterns of the 10 participants. Five participants (P1, P2, P5, P7, and P10) typed primarily using two fingers at the same time. Three participants (P4, P6, and P8) used multiple fingers on both hands at the same time. One participant (P9) only used only one finger when typing. Another participant (P3) changed the typing pattern over time (starting with only the left index finger, but adding fingers as the usability evaluation went on) and primarily used the left index finger but typed pretty fast on the virtual keyboard (15.8 words per minute). P6 and P8 both used multiple fingers on both hands to type at the same time, but their typing speed was below the group average of 10.4 words per minute.

It is possible that there might be a disconnect between the typing pattern using the traditional physical keyboard and the touch-screen. Both P1 and P3 participated in a previous study examining the use of traditional keyboard and mouse for data entry. In that study, they both typed using multiple fingers on both hands. However, they only used one or two fingers when typing on the touch-screen.

Table 11. Typing Patterns of the 10 Participants

Table 11

The iPad Notes app had an auto-suggest feature for words, but it was difficult for the participants to understand how to use the feature, so most participants ignored the suggestions. Participants assumed that the auto-suggested word could be tapped or selected on the touch-screen like many other auto-suggested features of applications; however, this feature only works if the user selects the space key when the word appears. Figure 8 shows an example of an auto-suggested word.

Figure 8

Figure 8. Example of an auto-suggested word on the iPad Notes app

With the exception of P7 (who ranked the task as difficult), the users P1, P2, and P3 all ranked the task as easy, while all other users ranked the task as very easy.

 

Previous | Next