The Magazine of the Usability Professionals' Association
By Libby Hanna and Kirsten Risden
What is the typical reaction to your research reports?
“Wow, thanks for sending that report. Those findings are really important and now we know what we need to do.”
“Uh, gee, we’re still working through that report, thanks. However, we do have some questions about the data. Can you answer those for us now?”
If you sometimes hear the second response (and who hasn’t?), or you want to review your writing style to make sure it’s having the impact you desire, there are two key things to keep in mind:
Aren’t there existing standards for user research reports?
Yes, there are existing standards.
The usability community has an international standard, ISO/IEC 25062:2006 that provides a “common industry format” (CIF) for reporting usability test findings. The standard consists of a seventy-four-item checklist for what should be included in a report, and can be found at http://zing.ncsl.nist.gov/iusr/documents/cifv1.0.htm. While important and useful for thorough documentation and archiving, the CIF comes across as a dissertation, and most of us don’t want to read one. So let’s move on past the CIF.
Classic usability textbooks (for example, Rubin and Chisnell’s Handbook of Usability Testing, or Barnum’s Usability Testing Essentials), offer a mixed bag of report styles. They start by outlining how to write a formal report and then suggest different formats that might work better. As Rubin and Chisnell write, “Deliver your report in a slide deck, in a company blog, or even an email if that way works best in your situation.”
How are other researchers doing their reports?
A quick survey with 28 respondents, distributed via the chi-consultants discussion list, found the following:
Arguments in favor of the document-style report are that it leaves a comprehensive record of the research and satisfies an audience that likes a lot of detail.
Arguments for the slide deck-style include that it’s easier to write and consume, and that it provides better formatting for visuals. Your decision on what style to choose will ultimately come down to your audience:
PowerPoint has become one of our most requested, and, we believe, most effective styles of reporting. Why?
Create your PowerPoint report in two parts: the first two-to-three slides are the executive summary and the rest are the usability issues list. The list is especially effective when you use screenshots to point out the issues and explain recommendations.
Does PowerPoint automatically make information glanceable?
No, it does not.
Some business executives believe PowerPoint presentations should consist of no more than three slides. Others believe all content should fit into one slide. The danger in that case is that you can end up with a very busy slide (see Figure 1).
The slide in Figure 1 requires careful reading of the summary, tracking numbers back from each cell, and guessing why some are highlighted. This works directly against one of the most appealing features of PowerPoint—its flipability. Readers should be able to flip through slides quickly to get to what is relevant to them. Therefore, you should keep the information clear and concise.
What does it mean for a slide to be “glanceable?”
A glanceable slide is one that uses callouts or pointers to show what’s important, rather than describing it in figure captions or paragraphs (see Figure 2).
The Executive Summary
An executive summary starts with the basic gist, and then expands as necessary. What’s the basic gist? Think:
The executive summary is the most important part of a report because it’s the only part that most people will read. It needs to stand in for the entire report. In discount usability testing, it will be the entire report in an email.
Don’t spell out methodology here. Your reader will generally assume you know how to design a good study, and won’t want to get bogged down by details. Focus on what you learned, and distill those findings down into the briefest possible phrases and points.
How can I draw my readers in right at the start?
Open with a quote.
What better way to communicate results of research with real people than to quote one of them right away? The best examples are the juicy quotes that exude love and joy for the product design or express frustration over poor usability. Picking one or two quotes from participants to start off a report brings your results to life and gives your readers an immediate sense of the validity and importance of the research. For example:
“I’m shocked at the things I was able to do...(my old program) took me several weeks to figure out some basic stuff…with this I could feel my way around right away...”
How can I get my readers invested in reading the whole thing?
Pull out some positives up front.
Everyone likes praise. Most usability studies will find some things that are working just fine, but there is a tendency to ignore those in favor of things that need to be fixed. We’ve found that opening with “highlights” or “things working well” helps to ease the tension and lower defenses.
Even in a straightforward usability issues list, we include a priority category of “good” or “working well” to make the point that certain UI elements are well-designed and should not be changed. This is especially important if the positive findings are answering research questions.
A good way to organize a document-style report is to use a question-and-answer format. It’s an easy-to-read, conversational style that helps readers locate the information most relevant to them. For the designer who created the blue dot display, what better way to get her the data she needs than to have a heading, “Did people notice and use the blue dot display?”, followed by the quick answer “Yes, they did.” A sample format would look like this:
How can I make my report glanceable?
Add a Table of Contents at the start, created from section headings. Readers will see at-a-glance what the report contains and be able to jump to whatever section they are interested in reading.
If I have to include numerical data, what’s the best way to present them?
Graph the data.
Tables may be a great way to organize data, but they’re convoluted and take time to decipher. Graphs are better; because readers can immediately see comparisons. Include the actual numbers in the graph and use error bars to show confidence intervals or standard error of the mean, and you will satisfy even the most statistically savvy audience (see Figure 3).
The tips we have found most effective for generating interesting and usable research reports are:
Libby Hanna is a research psychologist with more than fifteen years experience working as a consultant in usability research, currently at Hanna Research & Consulting, LLC. Her clients include Microsoft, T-Mobile, Intuit, and small start-ups, and projects range from extensive benchmark testing, to longitudinal field research, to quick-and-dirty one-day usability tests.
Kirsten Risden is a former user research manager for Microsoft Corporation and now has her own consulting firm, User Research Associates. Her projects include international research on educational systems in China and the U.S., and usability testing on touchscreen interfaces, e-commerce websites, and product tutorials.
Usability Professionals' Association
promoting usability concepts and techniques worldwide
User Experience Magazine is by and about usability professionals, featuring significant and unique articles dealing with the broad field of usability and the user experience.
This article was originally printed in User Experience Magazine, Volume 11, Issue 1, 2012.
© Usability Professionals' Association
Contact UPA at http://www.usabilityprofessionals.org/about_upa/contact_upa.html