Unexpected Complexity in a Traditional Usability Study
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205
Abstract
This article is a case study of a demonstration project intended to prove the value of usability testing to a large textbook publishing house. In working with a new client, however, the research team discovered that what our client thought were simple problems for their users were actually complex problems that required the users to evaluate potential solutions in a surprisingly complex context of use. As Redish (2007) predicted, traditional ease of use measures were "not sufficient" indicators and failed to reveal the complex nature of the tasks. Users reported high levels of satisfaction with products being tested and believed they had successfully completed tasks which they judged as easy to complete when, in fact, they unknowingly suffered failure rates as high as 100%. The study recommends that usability specialists expand our definition of traditional usability measures so that measures include external assessment by content experts of the completeness and correctness of users' performance. The study also found that it is strategically indispensable for new clients to comprehend the upper end of complexity in their products because doing so creates a new space for product innovation. In this case, improving our clients' understanding of complexity enabled them to perceive and to take advantage of a new market niche that had been unrealized for decades.
Practitioner's Take Away
- With new clients usability specialists need to accommodate the likelihood of encountering complex problems masquerading as simple ones and ensure that the studies we design use sound methodological triangulation techniques, including content experts’ assessment of the quality of the users’ performance and, when possible, head-to-head comparisons with competing products.
- Users in this study assumed that they were dealing with a simple problem, and once they found what they thought was the simple solution, they didn’t look any further for more complex answers.
- Users gave a positive evaluation of the ease of use for products that they believed had helped them complete a task when, in fact, the information products misled them, allowing them to believe that they had finished tasks that were only partially completed.
- To avoid giving users a false sense of success that can ultimately lead to poor user performance, visuals must signal to users that they are dealing with a complex problem.
- Maintaining a singular purpose in visuals is important for success, even though complex information needs to be delivered. Use several visuals to convey different purposes.
- Users need tasks that help them understand where to begin the decision-making process. When users understand the role they are supposed to play, they also understand the logic they’re supposed to follow in order to make the decisions necessary to complete the tasks at hand.
- Modeling complex problem solving behaviors through the use of scenarios may lead to more effective performance from users when it’s simply not possible to capture or replicate all of the potential variables in a situational context (Flowers, Hayes, and Swarts 1983).
- Helping new clients understand the upper end of complexity in their products can make it possible for designers to perceive new opportunities for product innovation and can help them create new strategies for market success. In this case, helping the client see that their competitor had failed to recognize the complexity involved in citing MLA sources and in responding to commas also helped them find innovative ways of entering a market that had been dominated by one product design approach for decades.
Article Contents
Download Article
Unexpected Complexity in a Traditional Usability Study