A Usability Evaluation of Workplace-Related Tasks on a Multi-Touch Tablet Computer by Adults with Down Syndrome

Libby Kumin, Jonathan Lazar, Jinjuan Heidi Feng, Brian Wentz, and Nnanna Ekedebe

Journal of Usability Studies, Volume 7, Issue 4, August 2012, pp. 118 - 142

Article Contents

Methods

We decided that for the current investigation of using touch-screens for workplace-related tasks, a modified usability testing methodology would be the most appropriate approach. Usability testing, at a basic level, is when representative users perform representative tasks (Nielsen, 1993). And a typical focus of usability testing is finding and fixing flaws in an interface. This research study is designed to be a preliminary investigation of how users with Down syndrome could potentially utilize touch-screens for workplace tasks to obtain a sense of some of the potential challenges to effective use of tablet computers for this population and to investigate how usability testing involving people with Down syndrome could be effectively performed. Typically, a usability testing method is used to understand what improvements are needed in interfaces (or compare the effectiveness of different interfaces), whereas experimental design is utilized more often to study users themselves or interfaces (controlling one aspect and studying the other). The focus of this study is equally on understanding the users themselves and also understanding potential interface improvements. Usability testing is the most appropriate method to reach this goal for the following reasons:

There is very limited research on computer usage by people with Down syndrome and none on touch-screen usage by people with Down syndrome. It is difficult to use experimental design for preliminary investigations like the current study when the relevant factors are not even completely identified. So we are interested in understanding the issue from multiple different viewpoints, with a more structured method than ethnography, but less structured or focused than experimental design. This study helps identify key factors that are worth further investigation through future experimental studies.
Experimental studies usually require full control of one or more relevant factors. However, there were many factors in this study that would be impossible to control, such as the specific, personal email account used for the email tasks. In an experimental study, it would be expected for all participants to use exactly the same email account (e.g., Gmail or Windows Live). But for users with Down syndrome, it was important to use their own email and Facebook accounts, because security and privacy are major issues in the population of computer users with Down syndrome. Parents have serious discussions with their adolescent and adult children about these issues. Using a fictitious account (as is typically done in experimental design and often is done in usability testing) could be viewed by participants with Down syndrome as using someone else’s account. It would feel dishonest and wrong for the participants to use someone else’s account. Furthermore, using a participant’s personal accounts, when possible, increases the validity of a usability testing session by engaging the participants more deeply in the session (Zazelenchuk, Sortland, Genov, Sazegari, & Keavney, 2008). However, using each individual’s own account presents problems for a true experimental design. Each participant’s Facebook account would differ due to the number of friends, number of pictures posted, and whether the account uses the Facebook timeline feature. Every user could potentially have a different email account from a different provider (Gmail, Yahoo! AOL, Comcast)—each with a different interface.
Because touch-screens and tablet computers differ so greatly, both in terms of their screen interface as well as the gesturing required, it would be hard to generalize many of the results from one tablet computer (such as the iPad) to another (such as the Samsung Galaxy). Our goal is primarily to understand usage of the iPad, not all tablet computer devices, and to explore whether adults with Down syndrome are able to use the iPad successfully. Only a few of the findings (related, for instance, to touch-screen-based keyboarding) could be generalized to other devices. Usability testing is similarly focused on understanding one interface, not on finding statistical differences and generalizing to many other interfaces, as is common for experimental design.

Pilot Sessions

Two pilot sessions took place before the research method was finalized. First, an observation was made of a 28-year old female with Down syndrome, who showed us how she uses technology involving touch-screens. There was no formal data collection or methodology, only an observation of her usage patterns. After the observation of the 28-year old female, the task list was drafted and a methodology was developed. A second, more formal pilot study was made, involving a 23-year old male with Down syndrome attempting to perform the tasks on the initially developed task list. Modifications were made to the wording of the task list as well as some of the tasks themselves. For instance, one of the original tasks was to have the participant accept a “friend request” on Facebook, but we learned that due to the high privacy settings that many people with Down syndrome have—they often cannot receive friend requests without parental permission. We also added a “warm-up task” to help the participants get comfortable with the researchers and with the iPad. The warm-up task was selected to be a weather related task where participants must find out the local temperature using a Web site. Also, a question about whether any computer classes had been taken was added to the demographic questions at the beginning of the data collection.

There are different approaches to moderator involvement in usability testing. For some types of usability testing, the researchers rarely assist or speak to the participants during the usability testing session. In other types of usability testing, participants are verbally encouraged by the moderators or given a hint to allow them to move to the next step in a task, but at no point are told what to do or how to accomplish the task. In yet other forms of usability testing involving one large task with multiple sub-tasks, if a participant cannot perform a task and asks for help, the researchers will perform an “intervention” that is well-documented in their notes (Lazar, Olalere, & Wentz, 2012). Based on our experience from the pilot studies, in the current research project, the participants received verbal encouragement from us and infrequently received a hint for how to move forward one step. However, at no time did we tell the participants specifically what steps to take or perform an intervention (Dumas & Loring, 2008). Participants were encouraged to think aloud as they were attempting their tasks, but the level of think-aloud varied depending on the personality of the participants. While parents or support people for the individuals were allowed to stay in the research room if they preferred (some did and some didn’t), at no point were they allowed to tell the participants how to do anything, with one exception: when participants had trouble remembering their passwords (as mentioned in later sections of the paper), parents or support people were allowed to provide the passwords. The sessions typically lasted between one and three hours, usually with a short break half-way through the tasks.

Participants

Ten participants took part in the study. All participants were required to meet the following criteria:

Be at least 18 years of age.
Have the Trisomy 21 form of Down syndrome (the most prevalent form for 95% of the population)
Have previous experience with computers and the Internet.
Have a minimum of basic experience with touch-screen computers.
Have an existing Facebook account and a Web-based email account that they could use for the study.

Demographic questionnaires were sent to the participants or their parents prior to the study so that they could have sufficient time to think and answer the questions. The demographic information of the 10 participants is listed in Table 1. Note that students receiving special education services in the US can attend high school until 21 years of age, which explains a number of participants who are 19 years old and in high school. The participants ranged from 19-29 years old (average age: 22.9 years old), and there were four men and six women.

Table 1. Demographic Information for the 10 Participants

Table 1

Table 2 lists the computer experience for the 10 participants. For instance, only four of the participants own their own multi-touch device, even though all of the participants have used multi-touch devices before. The experience on Web usage ranged between 2-15 years, and the average number of hours per week on the Web ranged from 2-30 hours per week. Some participants preferred using an iPad, some preferred a laptop computer, and others preferred a desktop computer.

Table 2. Computer Experience of the 10 Participants

Table 2

Table 3 lists the level of experience with email and Facebook. As is typical for participants with Down syndrome, there is a wide variance of experience levels. Some of the participants had only 2 years experience with email, while other participants reported as much as 13 years of experience. Some participants reported checking email once a day; while one participant reported that they checked email as often as 30 times a day. And one participant stated that they very seldom check Facebook, while another participant reported spending 12 hours a week on Facebook (which would average out to at least an hour and a half per day).

Table 3. Participant Experience with Email and Facebook

Table 3

Table 4 reports the participant experience with applications such as electronic calendaring, purchasing items online, and computer classes. For instance, seven of 10 participants reported having taken formal computer classes, which is common for non-introductory computer users with Down syndrome. To fulfill all of the inclusion requirements in this study (experience with touch-screens and your own email and Facebook account), it is likely that a participant would have taken a formal computer class. While these participants are by no means expert users (the Lazar et al. 2011 study on expert users with Down syndrome required a much higher level of skill and weekly usage), the participants in this study were not likely to be introductory users. This was not a random sample of people with Down syndrome, rather this was a group of people who were interested in the topic of tablet computers and who met the inclusion requirements. Random samples of computer users with disabilities are almost impossible to implement due to the lack of directories of users with specific disabilities and the availability and/or willingness of people with disabilities to take part in research studies (Lazar et. al. 2010). In any HCI study involving people with disabilities, it is likely that people who choose to take part in a research study have an interest in technology. Therefore, this study is likely representative of people with Down syndrome who are interested in technology, not representative of all people with Down syndrome.

Only four of the 10 participants had previously used an electronic calendar. Of the 10 participants, three had independently purchased something online, although six of the participants had previously done comparative price checking on various Web sites.

Table 4. Participant Experience with Computers, Online Shopping, and Computer Classes

Table 4

Tasks

Participants were asked to perform tasks in five different categories on the iPad. Five categories of tasks that are typically important for computer usage in the workplace were selected: social networking, email, calendaring/scheduling, price comparison, and basic text entry/note-taking. Social networking and email are forms of communication, and social networking has had increasing usage in the workplace. Updating a corporate social networking site is often a task assigned to certain employees. Calendaring and price comparisons are forms of information retrieval. Calendar usage and price comparison are possible work tasks that might be requested of an individual in an office assistant role. While the challenges and successes of people with Down syndrome related to keyboarding have been previously documented (Feng et al., 2008; 2010), there is no documented data on the use of a touch-screen-based keyboard interface (virtual keyboard) for text entry/note-taking tasks (as discussed in previous sections), and this was an important part of the data collection. The tasks under each category are listed below.

Social networking tasks

Use the Safari Web browser, go to www.facebook.com, and type in your email address and password.
Search Facebook for the page for the National Down Syndrome Congress. Click to “like” the page.
From your Facebook home page, under Messages, find a friend on Facebook (they don’t have to be logged on to Facebook right now) and send them the following message using Facebook: “What is your favorite restaurant? Mine is [user’s favorite restaurant was included here].”
Find the most recent status update (recent activity) for any friend of yours on Facebook.
Logout from your Facebook account.

Email tasks

Using the Safari Web browser, login to your email account, such as Gmail, Yahoo Mail, or AOL.
Create an email message saying “Hope you have a good weekend, [researcher’s name was used here]!” and send it to [researcher’s email address was used here]. You can leave the subject line blank.
Add a person to your email account address book/contact list. The person’s name is [particular name was used here], who is the head of public relations at [particular organization was used here], and her email address is [email address was used here].
Using your address book/contact list, send a new message to [individual’s name was included here] (with no subject in the subject line) and have the message say, “Hope you have a good weekend, [name was used here]!”

Calendar tasks

What is the date of the World Down Syndrome Day (it is on the calendar)?
Please select the day of your birthday on the calendar, and then “add event” with a title of “birthday party” starting at 12:00 p.m. and ending at 2:00 p.m., with a location of [user’s favorite restaurant was inserted here].

Price comparison tasks

Go to the barnesandnoble.com Web site. How much does it cost to purchase the new paperback book in English titled “Count Us In: Growing Up with Down Syndrome,” written by Jason Kingsley and Mitchell Levitz?
Go to the amazon.com Web site. How much does it cost to purchase the new paperback book in English titled “Count Us In: Growing Up with Down Syndrome,” written by Jason Kingsley and Mitchell Levitz?
Go to the staples.com Web site. How much does it cost to purchase the Olympus VN-8100PC Digital Voice Recorder?
Go to the officedepot.com Web site. How much does it cost to purchase the Olympus VN-8100PC Digital Voice Recorder?

Participants were asked to compare the price of each pair and point out which Web site offers a lower price.

Text entry/note-taking task

There were three slightly different versions of the note-taking text used. P1 and P2 used the following text:

The weather will get much colder, so I need to remember to wear my hat and gloves. I’m going to meet my friend at 4:30 p.m. to go to the new movie “War Horse.” Then I need to check the bus schedule to see if there is a different schedule on Lincoln’s birthday, which is Sunday, February 12th.

P3, P4, P5, and P6 used the following text, which was modified after the date for Lincoln’s birthday had passed to avoid confusion, replacing the date with Mother’s Day:

The weather will get much colder, so I need to remember to wear my hat and gloves. I’m going to meet my friend at 4:30 p.m. to go to the new movie “War Horse.” Then I need to check the bus schedule to see if there is a different schedule on Mother's Day, which is Sunday, May 13th.

P7, P8, P9, and P10 used the following text, which was again modified to avoid confusion about the cold weather because it was now warm, and the word “new” removed because “War Horse” was no longer a new movie:

The weather will get much warmer, so I need to remember to wear my shorts. I’m going to meet my friend at 4:30 p.m. to go see the movie “War Horse.” Then I need to check the bus schedule to see if there is a different schedule on Mother's Day, which is Sunday, May 13th.

To the outsider not familiar with people with Down syndrome, it may seem as if there would be no need to change the text. However, people with Down syndrome are often concerned with the accuracy and reality of details, so, they might object to writing text about an upcoming date that in fact had already passed. In later sections, an example is presented where P10 refused to enter an event into a calendar because it disagreed with the participant’s perception of what the data should be. It is important to note that the original text entered by P1 and P2 had a Flesch-Kincaide Reading Ease ranking of 80; the text typed by P3, P4, P5, and P6 had a ranking of 90.7; the third version typed by P7, P8, P9, and P10 had a ranking of 91.1. The Flesch-Kincaid ranking is a common way to assess the readability of text, and the higher score means that the text is easier to read.

Procedure

All participants received the instructions and tasks on paper, but also had the instructions and tasks read to them before they started each task. After participants completed each category, they were asked to rate the difficulty of the tasks using a 5-point Likert scale. Based on the pilot studies (and due to the visual strengths of people with Down syndrome), a visual Likert scale was created (see Figure 1).