upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Extremely Rapid Usability Testing

Mark Pawson and Saul Greenberg

Journal of Usability Studies, Volume 4, Issue 3, May 2009, pp. 124-135

Article Contents


Questionnaires

We originally planned on a short pre-questionnaire and an optional post questionnaire (e.g., a satisfaction or a desirability survey). We knew that time would be short in the booth and that participants would be eager to get to the product, so we wanted the questionnaire to be equally short. Thus we focused only on a few key questions that the company considered critical.

Athentech's previous research had already validated that Perfectly ClearŪ was aligned with customer goals. Their concern was with the offerings of recent competitive products on the market. Athentech felt that those products offered a different workflow and unnecessary functionality. Athentech also thought other vendors had understated the limitations of the digital camera in capturing true images. Given this, we targeted our pre-questionnaire to simple demographics (if participants were professional or serious amateur photographers), what software tools they were currently using for their work, and what they were using these tools for.

However, there were tradeoffs. Athentech also wanted to collect additional user feedback on various topics that would help guide their future software development. This would have dramatically increased the size of the questionnaire. We were concerned that customers would be turned off; they were drawn to the usability test (which was in the spirit of trade show demonstrations) but not to the barrage of questions both before and after the test. We found it challenging to balance the questionnaire so that it met both business and testing needs while respecting the customers' short timelines and interests. As discussed later in our "Lessons Learnt" section, flexibility was the best approach. Instead of requesting this extra information as part of the written questionnaire, we worked the questions into our conversation with participants while they were doing the task. We were opportunistic: we asked questions when they fit into the flow of activity, but in the interest of time not all questions were asked.

We also found that our post-test survey questionnaire did not work in the context of the booth. The questionnaire did not fit with the natural rapport of a trade show booth. As one participant said "everything you have done up to now has been great, but this just turns me off."

Choice of Tasks

We develop three tasks ahead of time that were both unique and representative of problems we believed that potential customers wanted solved and that incorporated the unique selling proposition of Perfectly ClearŪ. This was a modification of an idea used by Chauncey Wilson for testing in a trade show booth (personal communication, 2007). We had planned to let participants select the most personally interesting one of these three tasks to do. We thought the choices made would give us insight on what parts of the product the participant perceived as the most useful.

However, we decided that this approach was not the best one. First, the alpha release of Perfectly ClearŪ was not robust enough to allow people to actually do some of these independent tasks. More importantly, Perfectly ClearŪ was targeting a specific task workflow, cull and image correction of photos. Athentech was in part positioning itself against its competitors who (Athentech believed) had lost sight of this basic customer need by adding layers of complexity and functionality. Consequently, we decided to concentrate only on a core task that addressed this specific workflow. If that could not be done by people to their satisfaction, then it wouldn't really matter how well they could do other tasks with the system. Therefore we spent time with Athentech learning about the specific problems photographers faced with image enhancement and how this was addressed by Perfectly Clear'sŪ workflow. From this we created four interrelated scenarios in a photographer's language that we felt were both representational and motivational. These tasks were originally written down on 4 x 6 cards and were to be given to the participants as they completed each task. However, as in the questionnaire, we found the best way to introduce the task was as part of an informal conversation rather than by script. Hence the exact language used to introduce each of the four tasks varied between participants.

While the above may sound like normal task selection and debugging, we want to stress that the short time line forced us to reconsider our tasks. We would likely have time for people to do only a single task, and we needed to ensure that the results were extremely practical.

Previous | Next