upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Usability Testing with Real Data

Alex Genov, Mark Keavney, and Todd Zazelenchuk

Journal of Usability Studies, Volume 4, Issue 2, February 2009, pp. 85-92

Article Contents


The Challenges of Real Data Testing

Incorporating users' real data into a usability study also poses a number of challenges. Based on the authors' experiences across multiple product teams at Intuit, there are three primary challenges to incorporating users' real data into a usability study.

The remainder of this paper examines each challenge in turn, drawing on sample projects across product teams.

Recruiting Participants and Getting Their Data

In order to conduct a real data study, one must first determine how to obtain participants' real data. Depending on the nature of the product, this can be as straightforward as asking recruited participants to bring their data with them to the study to as complex as partnering with a third party to extract user data files from multiple backend systems prior to the study, followed by mapping and uploading those data into the prototype to be tested. The following are some methods that the authors have used for obtaining real data and the situations where researchers might use them:

These methods vary widely in difficulty, but each has at least some issues. Having participants login to their own account or bring their own data to the study are logistically quite simple, but involve some recruiting and attrition challenges. People may be reluctant to share their details with others, especially when the product being tested involves sensitive information such as personal financial or medical information. Thus, recruiting becomes more difficult as recruiters have to approach a larger number of people before they acquire a sufficient pool of participants, and the recruitment costs for the study will typically increase.

Depending on the application, people may also have trouble finding or collecting the data to bring with them. In the case of studies done with Intuit TurboTax, this was not as much of a problem as it might have been, because even though the program requires user data from multiple sources, the users were already in the habit of pulling their tax data together to either complete their own taxes or to share them with an accountant. But other real data studies, such as some done for Intuit's payroll software, could not be done in the lab simply because too many participants failed to bring everything they needed and instead had to be done in participants' offices.

Recruiting participants to send their personal data to the research team in advance of the study can prove to be somewhat more challenging than having participants bring their personal data to each session. This approach takes participants' privacy and security concerns to a new level. With the growing fears of identity theft and phishing scams, people are increasingly wary of sending their information to someone they do not personally know. As a result, research teams need to take extra care and precautions to ensure the security of data transmission and handling (see the Security and Privacy section) as well as reassure participants that their information is secure. Another challenge of this method is incorporating the data into the usability study prototype. In a real data study evaluating Quicken Health, participants were asked to mail in their data in advance of the study. This approach required extra time and significant manual entry into the prototype to be tested. Additional time was needed to allow for engagement from the Quality Assurance (QA) team who helped with ensuring the quality of the data entry prior to testing.

In the case of obtaining participant data ahead of the study via a third party partner, the logistical issues grow exponentially. Extra steps required by this strategy include convincing the third party of the cost and benefits associated with using real data, informing potential participants of the real data process and getting their permission to obtain their information, setting security and privacy procedures with the partnering group, and so on. In the case of the FinanceWorks project at Intuit, the application was intended to operate from within a financial institution's Web site, so the most useful method for obtaining customer's real data was through the customer's bank. Finding a bank willing to participate in this test, and then overcoming the legal and organizational hurdles to obtain permission both within Intuit and within the bank to run this study, took over two months.

However, the richness of using the customer's bank data in the tested prototype made the time spent well worthwhile (see the Benefits sections) and the success of the study established procedures and a relationship that opened the door for more real data testing, both with that financial institution and others.

Previous | Next