upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Unexpected Complexity in a Traditional Usability Study

Tharon W. Howard

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205

Article Contents


Using punctuation scenario

In this scenario, users were asked to identify comma errors in a paragraph and to provide the page numbers from the handbooks that provided information about the correct comma usage. We presented users with a sample student essay pregnant with comma errors that were based on the 20 most common errors found in 3,000 college essays by Connors and Lunsford (1992; see also Smith, 2006). We identified four potential comma problems for the users and then asked them to locate information in the handbooks that told them:

  1. if a comma was required at the location indicated,
  2. if no comma was required, or
  3. if the comma was optional.

This allowed us to collect data on how easy it was for users to locate specific comma usage information.

In part one, users were asked about comma usage in the places indicated by the four numbered circles in the following paragraph:

In America1 it is quite possible to live in a cocoon2 oblivious to the world around you. Confined living situations and close-knit social structures can prevent an individual from ever experiencing a reality,3 outside of his or her own. Through an artistic medium such as photography one can get a glimpse of a world far removed. Gordon Parks photographer, artist4 and writer, was a liaison between those Americans in one world and their fellow citizens who subsisted in a completely different one.

In part two, a similar paragraph was used for the second handbook. After completing each part of the scenario, users were asked to rate the ease of use for each handbook using the same ease of use scale as the citing sources scenario.

Evaluating sources scenario

The third and final scenario gave users a research paper prompt that included both a research topic and a specific audience for the paper. Users were then given possible sources for the research paper and asked to use the handbooks to indicate if the source was acceptable, unacceptable, or if more information was required. Users were again instructed to provide the page numbers from the handbook that enabled them to make their determinations. And once again, we collected ease of use data on this task. However, while the data from this third scenario were of interest to the client in terms of making recommendations on how to improve their textbook, the focus of this article is on complexity, and most of the complexity issues resulted from the citing sources and comma usage scenarios.

Controlling for print quality bias

Because the scenarios required that the students use and compare both handbooks and because one text was an unfinished draft, care was taken to ensure that texts used were of comparable finished quality. The materials used in the study were color copies of Longman's forthcoming DK Handbook and color copies of excerpts from Diana Hacker's A Writer's Reference (2006). Efforts were made to provide equivalent sections of Hacker's text so that neither text users received was complete. Both texts were cut to size, printed on facing pages, and plastic-comb bound. The texts were divided into the three sections, separated by Post-itŪ note tabs that were labeled Commas, Eval. Sources, and MLA. Throughout the study, the test administrator referred to both of the texts as "prototypes" under development and did not reveal to participants that the Hacker text had already been previously published. To further disguise the fact that one handbook was an unfinished draft, while the other represented excerpts from a published text, the test administrator labeled each copy with initials only. Throughout the study, the Hacker copy was referred to as the HC handbook or the HC prototype. The DK Handbook was only referred to as the LP handbook or the LP prototype.

Post-test interview

Having used both handbooks to complete tasks involving MLA documentation, comma usage, and evaluating sources, participants were asked a series of questions that allowed them to reflect critically on their experiences. Users were asked to compare the handbooks, to make recommendations for improving the handbooks, and to indicate which of the handbooks they would recommend to their teachers and why. Data obtained from the post-test interview as well as the scenarios are discussed below.

Previous | Next