upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Extremely Rapid Usability Testing

Mark Pawson and Saul Greenberg

Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 124-135

Article Contents


Lessons Learnt

While every trade show and usability testing needs differ, we offer the following lessons learned for others to consider within their context.

Easy access to domain experts and potential customers. Perhaps the biggest advantage of ERUT over a standard usability testing methodology is the ease of access to a large variety of domain experts and potential customers in one place. There is no time spent recruiting participants, dealing with the logistics of scheduling, or losing time due to no-show participants. These issues simply do not exist. A trade booth, if designed well, is a natural attractant for people. People are at a trade show because they want to be, and they come into a booth because they are interested in the product. Recruiting these people as study participants is just a matter of suggestion.

Business comes first. In a trade show environment, the business need comes first. Most companies enter trade shows for marketing, not for testing. More importantly, trade show participants are there to see products, not to test them. Thus one should not expect to do rigorous usability testing in such an environment; incomplete questionnaires and tasks are the norm, and participants may shift their attention to their personal needs vs. keeping strictly to the test regime. Yet this shift of attention is also an opportunity, as it creates a type of contextual interview around the topic of user and business needs while running the test task (it is contextual in the sense that the trade show offerings are often part of the conversation). In fact, our experience from this was that a trade show booth might be the next best thing to observing users in the context of their real environment because they are there for themselves, seeking real solutions to problems they have, and they are primarily in the booth for their own personal gain. The result is very rich customer input on their needs.

Casual conversation over scripts and questionnaires. The best way to engage participants was to drop the usability script and questionnaires; we used casual conversation instead. In our case, participants had a real need for automatic batch correction of their photos. Event photographers in particular were in the booth because they wanted to know how Perfectly Clear® would save them time doing hundreds of image corrections and allow them to get back to their jobs-shooting photos. They were captivated by the message that they had heard from the Athentech representative and were keen to see the software. Introducing ourselves by giving the standard "thank you for participating in our usability test…" patter and then presenting them with consent forms and a pre-test questionnaire was cold and robotic and did not fit the pace of action. Instead we worked both the business questions and the task into an exploratory conversation. This immediately engaged them, showed respect for their time, and worked with the natural flow of a trade show environment. Participants wanted to talk shop, not be treated as a test subject. They were there to get answers, not to be asked questions. By being very familiar with the questions we wanted to ask, we looked for opportunities to introduce them as part of a conversation during the testing. This was probably the greatest value of the questionnaires-they became our talking points. The questionnaires helped us pick up on important points made by the participant that otherwise could have gone unnoticed unless one is a domain expert in photography. Of course, this comes at a cost, the loss of a script means that the process is not as repeatable. Different words (and different evaluators) may motivate people differently and large chunks of the script may be omitted. This also implies that the collected data is better seen as samples rather than a consistent outcome based on repeatable instructions and tasks.

Tasks need to be meaningful. The actual tasks done by participants and how they are introduced may also deviate from the script. The trade show setting meant that we needed to introduce the task in a way that was meaningful to the participant. In one case, we had a pair of participants who were looking in detail at a photograph and expressed a desire to make the red colors "pop out." Perfectly Clear® corrects photos back to the true colors; artistically enhancing colors (typically done using other products on the market) is not a feature. However, the software does offer an export function. Thus we changed our task on the fly to fit the participant's expectations and workflow. Originally, our final task read, "Now you have completed your enhancements, pick your three best photos and store them as high quality JPEGs in a folder of your choice on your computer." We turned their comment around and simply asked them, "How would you get that photo into the software of your choice to pop out that red?"

Previous | Next