upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Unexpected Complexity in a Traditional Usability Study

Tharon W. Howard

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205

Article Contents


In "Expanding Usability Testing to Evaluate Complex Systems," Redish (2007) observed, "Ease of use-what we typically focus on in usability testing-is critical but not sufficient for any product. Usefulness (utility) is as important as ease-of-use. If the product does not match the work that real people do in their real environment, it may be an easy-to-use solution to the wrong set of requirements" (p. 104). This study provides empirical evidence that validates Redish's observation. The traditional ease of use indicators in this study, both quantitative and qualitative, suggested that users found the prototype handbook my colleagues and I were testing to be an "easy-to-use solution" to their perceived needs. What's more, our client was entirely satisfied with the ease of use data we provided and didn't require that we use additional methodological triangulation in the study. As is often the case with clients new to usability studies, the clients were more concerned with "whether users liked using the product" than they were with what the study might be able to tell them about accuracy and functionality. This common attitude among clients who are new to usability studies can make it difficult for usability professionals to justify the additional expense and complexity of research designs that collect data on "utility" as well as ease of use (Redish 2007). However, had this study's design solely focused on what the client told us they wanted and had we only used traditional measures, we would have failed both our client and the users. Consequently, this case will hopefully serve as an argument that can be used to show clients why we need to go to the additional expense of methodological triangulation. What's more, this case is also a "success story" that can hopefully be used to illustrate how helping clients understand complexity they didn't know existed in the use of their products can create a space for innovation and creative thinking. In other words, this is a story about how discovering unexpected complexity enabled our client to release what has become a tremendously successful product into a market that had been dominated for decades by a traditional type of product design.

Definitions of complex problems vary, but according to Michael Albers (2003), "In complex problem solving, rather than simply completing a task, the user needs to be aware of the entire situational context in order to make good decisions" (p. 263). In doing a comparative usability test of two handbooks for students in freshman composition classes, this study provided a very interesting case in point of Albers' statement. Students thought the writing handbooks were easy to use, but they chose incorrect solutions to problems, because the handbooks did not help them understand the complexity of their task and did not provide guidance on how to choose wisely within that complexity.

This case study is also particularly interesting because we worked with textbook publishers. The computer industry has long benefited from usability studies, but the textbook industry has traditionally relied on reviews by experts. They rarely take time to gather user experience information from the actual users of their products, i.e., the students. In this project, my colleagues and I worked with the Director of Research and Development for a major textbook publisher, Allyn & Bacon/Longman, to design a study that sought to demonstrate that usability testing of textbooks with actual users could be used both (a) as a means of helping authors to produce more usable books and (b) as a means of helping acquisition editors and senior management evaluate whether or not to invest the resources needed to take the product to market.

Initially, the study had three goals:

However, as the study progressed and we learned more about our new client's understanding of the complexity of the tasks users were expected to perform with handbooks, a fourth goal emerged. We were able to demonstrate for the client that aggressively seeking to understand complexity in a product that they hadn't known was there simultaneously enabled them to see new market opportunities and helped create a space for creative thinking about a product that had been on the market for decades.

Previous | Next