upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Unexpected Complexity in a Traditional Usability Study

Tharon W. Howard

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205

Article Contents

Users failed to recognize the complexity of their situations

Admittedly, the books that users were tasked with citing were challenging, but a strength of the study was that the task was also realistic. One of the works users had to cite was a corporate author where the corporation was also the publisher. The other was a book chapter with three authors published in the second edition of an anthology edited by four people. It was necessary to challenge the users so that they would actually need to use the handbooks to complete the tasks, and it is the case that all the information needed to cite both of the texts is provided in both handbooks. The books were intended to support precisely this sort of challenging citation, so if the handbooks were to be considered truly usable, it seems legitimate to have expected that users should have experienced more success than the total failure we observed.

Furthermore, we saw similar performance issues in the responses to the punctuation scenarios. Although the findings were less problematic from a performance perspective, our study found that users consistently failed to correctly indicate when the use of commas was required, not required, or optional, and they also failed to provide the correct page number from the texts where they obtained the information. For example, 11 of the 12 users incorrectly stated that a comma was required rather than optional after short, 2-word introductory clauses, and once again, this finding was observed for both handbooks. However, the users were completely unaware of this deficiency in their performance and did not consider this factor when they assessed the "usability" of the handbooks. It was the discovery of these extremely poor performance indicators for both texts that initially led us to question what might have been at issue. Had only one text failed, then this might have suggested that it was the delivery technique used by that product that was at issue. However, the failure of both products was the first real clue that complex problems were at issue.

Why did users fail?

Unfortunately there is no single, obvious, one-size-fits-all explanation that describes why some users struggled with the comma sections in the handbooks or that can adequately illustrate why all 12 users failed to produce appropriate works cited entries. Several factors contributed to users' problems:

To illustrate the difficulties here, it may be worthwhile to examine the ways users attempted to address the question of whether or not a comma is required or optional after the phrase "In America" in the following sentence:

In America it is quite possible to live in a cocoon.

The correct answer to this question is that the comma is optional, which is explained on page 433 in the DK prototype and on page 236 in Hacker. However, only one DK user correctly gave "optional" as a response, and this user incorrectly identified the pages where the information could be found. All the other DK users incorrectly stated that a comma was required, but only 2 of those 11 gave page 432 as the page that indicated that a comma was required (see Figure 2 in the Background section). The other DK users gave page 428 as the page that contained the information because page 428 had examples of sentences that looked like the pattern. By contrast, all of the Hacker users correctly identified page 236 as the page with the information they needed, but only because the only examples available were on page 236.

Previous | Next