upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Extremely Rapid Usability Testing

Mark Pawson and Saul Greenberg

Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 124-135

Article Contents


Co-discovery, Think Aloud, and Active Intervention

We were concerned that the trade booth could create an intimidating atmosphere for usability testing. We did not know ahead of time how the booth layout would affect participant privacy and distraction, which in turn would hamper the concentration of a single participant asked to "think aloud" while completing a task. We decided to use co-discovery, where two participants work together to complete a task. Co-discovery yields higher quality verbal communication between paired participants than single participants. The pair typically converse for their own benefit to complete the task, as opposed to a single participant who is communicating solely for the test facilitator's benefit.

In the trade show context, we felt it unreasonable to pair strangers. Instead, we looked for people who visited the booth with a friend or associates and encouraged them to be our participants. Still, we did use single participants if no pair was around at the moment. In these instances, and given the predicted short test cycles, we used active intervention in order to elicit high quality think aloud comments. Active intervention was also advocated in a Web discussion forum on usability testing at conferences (IXDA, 2007). We were somewhat surprised at how well this worked. Only once did we have to ask a participant what they were thinking, all others proved textbook examples of the think aloud technique. We surmise that this is the result of the informality of our private testing area, the relaxed trade show atmosphere of the attendees, and participants' keen interest in the product.

In practice, we gleaned equally high quality think-aloud and co-discovery comments from both individual and paired participants. We certainly observed the engagement of paired participants with each other as the research has reported. However, we also found that it was quite common for one participant to break off his conversation and attention to the task. The participant would explain her thoughts to us or ask a question, while the partner carried on alone. We used active intervention on both the single and paired participants to work in guiding questions at appropriate times.

Storyboards for Recording Results

Recording test results in the fast pace, noisy atmosphere of the trade show raises other challenges. We used a modification of an HCI discovery technique described by McQuaid, Goel, and McManus (2003) to shadow and record the "story" of library visitors. They took pictures of the visitors as they pursued their activities. They printed these pictures and overlaid acetate sheets to record their notes of what they observed. Then, they compiled these into storyboards that they hung on a wall and displayed to stakeholders.

In a similar way, we used hardcopy screen shots of Perfectly ClearŪ to record the story of the paths the participants took in exploring the task. To clarify, storyboarding is a prototyping technique usually used to describe an interface sequence to others. Instead, we used storyboarding for note-taking, where the visuals and annotations described the primary actions a person actually did. We did not use videotapes or screen-capture software for recording the usability results, as we would not have had the time to revisit, analyze, and reflect on these recordings. As well, we were looking for high-level vs. detailed effects. It was unclear if video analysis was worth the effort. The advantages of paper storyboards are the ease in taking notes by simply circling or numbering areas visited, adding annotations as needed, and-perhaps most importantly-the immediacy of the result. The storyboards helped us collate our notes at the end of the day and perform our analysis without having to wade through hours of video tape. However, the storyboards are by no means neat, as annotations were made in a rushed pace. Notes on interesting observations, comments made by the participants, answers to questionnaires could all end up on a storyboard and these could be hard to decipher days later. Ours had to be looked at on the same day while our memories were still fresh in our mind. Also, unlike McQuaid, Goel, and McManus' storyboards, ours were far too messy to show to stakeholders.

Previous | Next