upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Reverse Engineering of Content to Find Usability Problems: A Healthcare Case Study

Shadi Ghajar-Khosravi, Flora Wan, Samir Gupta, and Mark Chignell

Journal of Usability Studies, Volume 8, Issue 1, November 2012, pp. 16 - 28

Article Contents


Conclusion

This study examined whether reverse engineering is a viable usability evaluation method in comparison to traditional task-based techniques. While the concept of reverse engineering stems from software engineering, in the context of usability testing, it can be used to identify tool design flaws that cause usability problems.

The results of this study suggest that the technique of reverse engineering has good potential for uncovering usability problems. Although the task using the reverse engineering method took a longer time for participants to complete, and with more problems uncovered, the participants’ attitudes toward the test process remained positive. An advantage of using the reverse engineering method was the simplicity in setting up the test—one simply needed to create a product using the tool that was to be tested and then ask a participant to recreate it using the same tool. One point to keep in mind is that reverse engineering usability methods require longer session times because it takes participants longer to perform the tasks.

Limitations

First, due to technical limitations, only one version of the Wikibreathe tool was used in the experiment. Because of this, each participant performed the task twice using the same tool and therefore the learning effect could not be measured properly. The task itself was also quite simple, and the study involved only 12 participants. Those participants were not respirologists, but were asked to imagine themselves in the role of a respirologist when performing the assigned tasks. For the purposes of this case study, the sample used was likely adequate to meet our goal of demonstrating the properties of the reverse engineering method of usability testing.

Further research is needed both with more participants and with different tools being tested in order to demonstrate the scientific validity of the reverse engineering method across a broad range of content and for different artifact generation tools. We had the opportunity to work with an important but some-what atypical context (asthma action plans) and it would be good to replicate the case study with a more mainstream user interface design context such as word processing or graphic design.

One final limitation was that one of the researchers acted as usability engineer in the usability testing, and thus the reporting of problems might have been biased in some way as a result. However, given that the set of problems identified were clearly defined and easy to characterize it seems unlikely that tester bias could explain the large increase in problems found when the reverse engineering method was done first. It also seems unlikely that possible tester bias could account for the SUS score that was obtained when the reverse engineering method was done first.

 

Previous | Next