Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study
Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell
Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28
Article Contents
Introduction
The purpose of this paper is to present a case study of the use of reverse engineering as a usability testing tool instead of the traditional task-based scenario method. We have heard anecdotally of usability specialists using reverse engineering in their practice, but there seems to be a lack of published discussions of the use of the technique. Thus, in this paper we present a case study in the use of reverse engineering for usability testing in the hope that it will promote increased use of the reverse engineering technique.
Healthcare is the area of application context for this case study, and more specifically the creation of asthma action plans. These plans are important artifacts that provide guidance on how to manage asthma as a chronic condition. We chose this context to explore the value of reverse engineering.
Background
Many methods for general purpose usability evaluation have been described (e.g., Sharp, Rogers, & Preece, 2011), both quantitative (e.g., measuring task completion time and number of errors) and qualitative (e.g., collecting feedback on user preferences through questionnaires and focus groups; Rubin, Chisnell, & Spool, 2008). However, one method that has received little attention is the case where a software tool produces a specific artifact, and participants are asked to recreate the artifact (using the tool) through a process of reverse engineering.
The term reverse engineering as it applies to software engineering is defined by Chikofsky and Cross (1990) as the following:
The process of analyzing a subject system to (i) identify the system’s components and their interrelationships and (ii) create representations of the system in another form or at a higher level of abstraction.
From a usability testing perspective, the four phases of a typical reverse engineering process(Storey, Sim, & Wong, 2002) can be adapted as the following sequential steps:
- Step 1–Parse: The test participant is given the final product of a system (e.g., a document or graphic) and identifies the elements comprising the product.
- Step 2–Analyze: Participant analyzes the elements of the product in order to understand how they are related to the features of the tool.
- Step 3–Document/Visualize: Participant looks at each element in more detail and maps them to the functionality of the tool. The participant may go back to Step 2 for further analysis of the elements.
- Step 4–Reengineer: Participant recreates the product using the tool while the observer notes any difficulties encountered during the reengineering process due to the design flaws in the system. Once the product is reengineered, the participant may go back to Step 1 and parse the product again until satisfied with the recreated product.
In the case study reported in this paper, we compared the reverse engineering approach to usability testing with a traditional task-based scenario technique. The research involved an evaluative case study using an online tool for creating asthma action plans. The tool was developed to let three different groups of stakeholders (respirologists, patients, and asthma educators/primary care physicians) create asthma action plans collaboratively. Further information on the development of the tool was provided by Wan (2009) and by Gupta, Wan, Newton, Chignell, and Straus (2011).
An asthma action plan consists of a set of written instructions on how to treat asthma when the condition is stable, as well as when the symptoms are active (Plottel & Feldman, 2008). Asthma action plans usually use the “zone” system, patterned after the red-yellow-green lights of a traffic signal, to describe the severity of asthma symptoms and to determine the appropriate response (Sloane, Slatt, Ebell, & Jacques, 2008). Studies have shown that the use of asthma action plans for self-management of asthma symptoms greatly improves the health of adult patients (Gibson et al., 2003), and in the case of asthmatic children, significantly reduces the number of acute care visits per child (Zemek, Bhogal, & Ducharme, 2008).
The case study reported here was part of a larger project named Wikibreathe (a.k.a. Online Collaboration Tool for Asthma Action Plan with Usability [OCTAPUS]). The primary goal was to both determine the preferences of different stakeholders for the design and content of an asthma action plan and to use an online tool to collaboratively author a consensus asthma action plan.