upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study

Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell

Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28

Article Contents


Tips for Usability Practitioners

We were interested in finding out if reverse engineering could be a useful alternative to task-based usability testing in this and perhaps other contexts. For a practitioner, would usability testing with reverse engineering be as easy to set up, administer, and analyze as a traditional test, and would it uncover a representative set of usability problems? Why should practitioners use the reverse engineering method when there are a number of methods already available that have a much more extensive track record?

Based on our experience, the reverse engineering method is worth a serious look. We found that participants had a positive attitude towards the reverse engineering method. Although the participants had to figure out how to carry out the task, the definition of the task seemed more straightforward and less arbitrary because they were trying to duplicate a meaningful artifact. Thus, we believe that the value of testing software using the reverse engineering method seems more self-evident to test participants than methods where they are told to carry out a series of tasks that have been defined for them.

We found that the reverse engineering method required more attention to detail by the participants, because of the requirement to make a copy of the target artifact. In some cases, this attention to detail might be beneficial in finding more problems, but in other cases it may find problems that are too detailed to concern most participants. Thus, they might put too much time and pay too much attention to some of the details that would not necessarily be confronted when using a task-based testing method. This is where practitioners have to exercise judgment in deciding whether or not reverse engineering may work for a particular tool and associated set of users. While it is possible that instructions to focus on the right level of detail might be beneficial in this regard, it was not an issue that we examined in our case study.

Our bottom line message for practitioners based on our experience with the case study is that the reverse-engineering approach is useful if you want to

On the other hand, the reverse-engineering method is not recommended if

Note that in any given study, the preferred approach might be to use a combination of usability testing methods. Reverse engineering could be used with other methods to allow participants to more fully explore products. However, based on our experience with the case study, we suggest that when reverse engineering is used along with other methods, it should generally be carried out first.

While this case study assesses the benefits of using reverse engineering for usability testing in a specific healthcare context, we believe that the benefits of the method found in this case study may apply more broadly.

 

Previous | Next