upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Unexpected Complexity in a Traditional Usability Study

Tharon W. Howard

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205

Article Contents


Scenarios and tasks for think-aloud protocols

After collecting this basic information about the participants, we introduced participants to the scenarios. Naturally, participants were instructed to talk out loud and to verbalize their thoughts as they attempted to use the textbook to perform the tasks provided. To help us track what they were observing on the pages, participants were instructed to point at the text as they moved through it and to read aloud when they were reading text. If users did not speak for more than 5 seconds, they were prompted by the test administrator and asked to explain what they were thinking. Also, active intervention protocol techniques were used to question users about their location in the text, what they were seeing, and how they felt about the material. Participants' comments were videotaped and coded, helping us to identify whether or not the page layout techniques, navigation systems, and other features of the text assisted or impeded users' ability to perform the tasks.

Our student users worked on three scenarios of increasing complexity:

  1. Putting a complex source into correct MLA style (the citing sources scenario)
  2. Identifying non-trivial comma errors (the using punctuation scenario)
  3. Evaluating the acceptability of sources based on information about a specific assignment and the audience for the piece they would be writing (the evaluating sources scenario)

Each scenario increases the complexity of the task to be performed. The citing sources scenario essentially asked users to follow a model in order to complete a task. Identifying non-trivial comma errors was slightly more complex because it asked users to apply rules to a situation and to make a judgment. And the final task, evaluating possible sources for a library research paper, was the most complex because it required an understanding of the rhetorical situation in which the sources would be used. Users had to make a judgment about the appropriateness of a source based on the exigency for the research paper, the audience for the piece, and wide variety of other environmental factors.

Participants used both handbooks for each scenario so they could compare the two handbooks. However, in order to control for first-use bias, the research team alternated which handbook they used first in each scenario. This ensured that both handbooks were used first an equal number of times.

Citing sources scenario

In the first scenario, users were asked to assume that they were working in their current composition class on a research paper. The scenario required that they create a works cited entry using MLA style. The researchers provided books with passages marked in them that users had quoted in the hypothetical research paper they had written.

During the pilot testing, we discovered that users would not actually use the handbooks thoroughly if the works cited entries were simple, single authored books. Because of their previous experiences writing research papers and creating works cited entries, it was necessary to challenge users with difficult and unusual citation tasks that actually required them to use the handbooks to find citation models that were unfamiliar to them. For part one of the citing sources scenario, the following text was used for the works cited entry:

Adobe Systems, Inc. ADOBE PHOTOSHOP 5.0: Classroom in a Book. San Jose, CA: Adobe Press, 1998.

For part two, users were instructed to use the other handbook, and the following text was used for the works cited entry:

Baecker, Ronald, Ian Small, and Richard Mander. "Bringing Icons to Life." Human-Computer Interaction: Toward the Year 2000. 2nd ed. Eds. Ronald Baecker, et al. San Francisco: Morgan Kaufman, 1995. 444-449.

At the end of each part, users were asked to rate the ease of use for the handbook using the following scale:

Very useful, Useful, Rarely useful, Not useful

Users were then asked to explain their rating.

Previous | Next