upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Extremely Rapid Usability Testing

Mark Pawson and Saul Greenberg

Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 124-135

Article Contents

The test requires a narrow focus on core issues. Focusing on core issues is critical, not only because time is short (Bauersfeld & Halgren, 1996; Millen, 2000; Thomas, 1996), but because it is likely those core issues will engage participants. Another advantage of the narrow focus is that it requires all stakeholders to define what the core functionality of the product is and what they hope to gain from usability testing in such an environment.

Interruptions are the norm. Even though participants were in a screened inner booth, interruptions happened and had to be accommodated. An example includes participants answering their cell phone. As well, some participants had to leave partway through the test due to conference talks or catching the last train home. Unlike a normal usability test, we could not expect people to set aside a fixed block of time solely for our purposes.

Participants perceive the test primarily as a demonstration. The trade show is a place to gather materials and see demonstrations. Even though we told people they were in a usability test, they still thought of it as an opportunity to try out the system, i.e., they did not really dwell on the fact that they were in a usability test. In one case, a participant responded to a cell phone call from a colleague by saying "Yeah, I'm in a demo right now. I want to buy this software, ok bye." To keep in this spirit, our final question was "would you buy this software?" As well, participants had the opportunity to sign up to get beta-releases of the system.

Tag teaming and active intervention. We found the best sessions were when the two experimenters were able to tag team each other rather than working alone. Although we tried working alone, there were times when note-taking disrupted the natural conversation with the participant. Key observations could have been missed, and the participant (whose time is precious) had to wait for the note-taker to catch up. Tag teaming allowed us to engage and disengage with the participant. One of us would write notes while another would pick up with a thread of interest. Tag teaming was a better fit to the trade show atmosphere, where we could engage participants in friendly conversation rather than sitting back quietly and watching. This active intervention by a team meant that participants were always being observed, that notes were being taken, and that they could talk to us any time.

Test time is variable. We originally felt that 20 minutes was the maximum time that we could expect from any participant. In practice, and somewhat surprisingly, most participants stayed much longer than that because they became engaged with the system. We allowed people to stay longer than planned when this happened. This also meant that strict scheduling could not be done. Instead, our "gatekeeper" would feed us participants as we were able to receive them.

Participant flow must be regulated. Because no scheduling is done, we needed some way to control the flow of participants into the test area. In practice, there were times in the booth where participants were let in too soon after a test has been completed, leaving us scrambling to get prepared (we needed about ten minutes between each test to collate our results, finish up any notes, and get the material ready for the next test). The problem was that the gatekeeper was busy with his own needs (marketing) and sometimes used the departure of a participant as an (incorrect) cue that we were ready for the next one. It would have been helpful to have had a green and a red flag by the doorway for the gatekeeper's benefit (red meant busy, green is ready for more test participants).

Previous | Next