upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study

Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell

Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28

Article Contents


A set of available asthma actions plans were surveyed to create an exhaustive list of features to be included in the Wikibreathe tool. Based on those features, an initial prototype was created. Then, a series of focus groups were conducted to gather initial feedback on the design of the Wikibreathe tool prototype. Based on the feedback collected, the revised prototype was developed and then evaluated using two usability testing methods: task-based and reverse engineering testing methods.


The following sections discuss the initial requirements for gathering data and the focus group that helped develop our prototype.

Initial requirements gathering

Before starting to design the Wikibreathe tool, available asthma action plans written in English were surveyed in order to create an exhaustive list of features that could potentially be included in the tool. Sixty-nine unique asthma action plans from around the world were analyzed for both content and format differences. Based on the requirements collected as a result of this survey, expert opinion, and best evidence, it was decided that the Wikibreathe tool should be able to generate an asthma action plan with the following sections:

Focus group

An initial prototype was developed based on the format and content features identified during the initial requirements gathering phase. The overall concept for the envisioned tool was that the user would choose from a list of options on the left side of the screen to build up the asthma action plan that would then be displayed on the right side of the screen. A series of moderated focus groups and interviews were then conducted to gather feedback on the initial design of the Wikibreathe tool. In order to clearly evaluate the preferences of the stakeholders and to avoid bias, each stakeholder group was evaluated separately.

The results of all three stakeholder groups showed that participants had a positive attitude towards the Wikibreathe tool. A major difficulty across all groups of participants was lack of proper organization in the navigation system. Based on the feedback collected during these focus groups, the initial prototype was modified, and a revised prototype was developed for testing using the two contrasting methods (i.e., the traditional task-based method and the novel reverse-engineering method). See Figure 1 for the revised prototype.

One point to keep in mind in reading the case study discussion is that while asthma action plans are used by patients, the Wikibreathe tool is designed for respirologists to make it easier for them to develop well-structured asthma action plans. Thus, the intended users of the Wikibreathe system were respirologists.

Figure 1

Figure 1. Revised prototype of the Wikibreathe tool

Usability Testing Methodology

This section discusses the process used to evaluate the revised version of the Wikibreathe tool using the two usability evaluation methods.

Participant selection

A convenience sample of 12 participants (4 Male, 8 Female; these participants were not the participants used in the focus group) between the ages of 20-40 were recruited from friends and family of the researchers. Their occupations ranged from a high school teacher and music therapists to a programmer and an engineer. They spent an average of 9 hours each week on the Internet at work and 10.4 hours each week on the Internet at home. All participants had a generally positive attitude towards computers. Thus, our participants tended to be well educated and computer savvy, as would be the respirologists who were the target users of the Wikibreathe system. If we were testing the effectiveness of the tool itself, a sample of respirologists would be needed. We were interested, however, in the testing method so we decided that for a case study, the sample we used would be appropriate.

User study process

Participants used the Wikibreathe tool with both a traditional task-based scenario approach and a reverse-engineering method. Participants were asked to use the tool to perform the same task of creating an asthma action plan twice (i.e., in a within-subjects design): once using the traditional task-based method and once using the reverse engineering technique.

During the user study sessions, participants were first introduced to the goal of the project and were told that the purpose of the study was to examine the usability of the Wikibreathe tool for designing asthma action plans. They were then given a brief introduction and training on the Wikibreathe tool. Each participant received the same amount and type of training. Lastly, they were shown three asthma action plan examples that we had created previously using the Wikibreathe tool.

Participants then used the Wikibreathe tool with one of the usability testing methods, then the other. The order of methods was counterbalanced such that six participants did the task-based method first and six did the reverse engineering method first. The following describes the methods used in this case study:

The process of preparing for the reverse engineering portion of the case study consisted of creating an asthma action plan, using the Wikibreathe tool, that participants could then attempt to duplicate. Thus, once we created an asthma action plan instance to use as the target in the reverse engineering method, the only remaining step was preparing a few straightforward instructions on how to do the reverse engineering task using the Wikibreathe tool (see Appendix A). In contrast, a detailed set of instructions and a moderator script were required as preparation for the task-based test method. A more important challenge in task-based usability testing is that the test administrator not only has to select which tasks to include and in what order but also to carefully word them so that they are clear and unambiguous. Besides, with task scenarios, it is almost impossible to probe all aspects of a product. Therefore, test administrators have to choose what to probe and that may limit their ability to uncover all of the important problems. These challenges are sidestepped in the “make what you see” approach of the reverse engineering method (Appendix A).

Because participants were to use the same tool with each evaluation method, we decided to use two different asthma action plans so as to reduce the learning effect of having to create the same plan twice. We constructed the two plans in such a way that they were different overall, but had similar tasks, so that the results were comparable.

During each user study session, the task completion time and percentage of errors made during each of the testing methods were recorded. Examples of errors included deselecting an option while the option had to be selected or not recognizing the different page layouts (Table 1). Participants were encouraged to verbalize their thoughts while using each method.

After each method, a short questionnaire was administered to the participant. The questionnaire consisted of the System Usability Scale (SUS) questions and a number of open-ended questions inquiring about the participant’s attitude towards the tool (for example, if they felt confident or frustrated while using the tool). The SUS questionnaire used a 5-point Likert scale where 1 was Strongly Disagree and 5 was Strongly Agree (Brooke, 1996).

After both of the methods were completed by the participant, a semi-structured interview was conducted. The interview inquired about the issues raised during the participant’s interaction with the tool. Participants were also asked about their overall impression of the tool.


Previous | Next