Usability Testing with Real Data
Alex Genov, Mark Keavney, and Todd Zazelenchuk
Journal of Usability Studies, Volume 4, Issue 2, February 2009, pp. 85-92
Abstract
Usability practitioners run the risk of misreading the results of usability evaluations, either identifying false positives when artificial user data interferes with a user's product experience or overlooking real problems when they use artificial user data. In this paper, we examine a strategy for incorporating users' real data in usability evaluations. We consider the value and the challenges of this strategy based on the experiences of product teams in a consumer software company.
Practitioner's Take Away
- Whenever possible, obtain your participants' real data by having them bring their data with them to the test and enter it themselves.
- If it's not possible to have participants either bring or send in their own data in advance, make sure to allow sufficient time to develop and test the technical and logistical process of getting the participants' data into your test prototype.
- Involve your company's privacy and legal stakeholders in decisions of how to handle the participants' data. Consider encrypting data, obtaining additional consent, and restricting access to videos as possible ways to safeguard the privacy of participant data.
- Prepare a tailored script for each participant by creating task templates ahead of time and then filling them in with each participant's data as they are received.
- When analyzing your results, consider constructing average scores for each participant on a per task basis before calculating success metrics across participants. This will help to reveal overall performances when not all participants performed all tasks.
Article Contents
- Introduction
- Real Tasks, Real Users … But What About Real Data?
- The Benefits of Real Data Testing
- The Challenges of Real Data Testing
- Conclusion
- References
Download Article
Usability Testing with Real Data