UX Magazine logoThe Magazine of the Usability Professionals' Association

A Clever Startup: Combining qualitative and quantitative UX research

By Stephanie Rosenbaum

What does a startup company like Spout (www.spout.com) do to learn how well their site is working for customers and investigate what would increase its success? Their website has been called, “A genius hybrid of MySpace and Netflix,” and Spout’s founders believe, “If we can make it easy for people to share movies they love with others…really meaningful stories will find their audience.”

Spout’s design partner, digital agency BBK Studios, recommended that Spout engage a neutral third-party research partner to help them understand how users interact with Spout.com. The designers at BBK Studios follow user-centered design methodology: they had conducted focus groups, a usability test, and ongoing user interviews to inform their design. Now BBK Studios judged it was time to involve user research specialists as they entered the next design phase for Spout.com.

Choosing an Integrated Approach

BBK was familiar with our firm Tec-Ed’s expertise in user research, and we began brainstorming in planning meetings with Spout. Unlike most startups, Spout and their design partner had laid out preliminary ideas for a year-long research program, so we had an excellent starting point for discussion. Like many startups, though, Spout had a limited research budget.

The Spout management team wanted to collect data with larger samples than had been used in the initial design research, so they first suggested that we conduct survey research. However, Spout’s primary goal was to gain more in-depth understanding of their potential user community, and several of their key questions asked why site visitors behave as they do.

We discussed the trade-offs with key stakeholders. A benefit of working with a small startup client is that Spout’s CEO and marketing VP were involved in the planning, as were BBK Studio’s management. Then we recommended a balanced program of qualitative and quantitative research that addressed Spout’s questions and stayed within budget. We proposed conducting qualitative usability testing twice a year, with each test to be followed by quantitative research to address questions that emerged from the qualitative research results.

Anticipated Challenges

During our early planning, we were concerned about achieving valid results from the quantitative research because even the most carefully crafted survey can only report what people say, not their actual behavior. Some surveys fail to predict user behavior because respondents are answering the questions outside their context of use, and no researcher is present to probe into the reasons behind their answers.

Therefore, Tec-Ed recommended improving the rigor—and success—of Spout’s quantitative research plan by conducting (ital)task-based surveys. Our goal was to introduce a behavioral component into the quantitative research phase by having participants perform tasks on the Spout website during the survey. This would require careful coordination not only between Tec-Ed and Spout, but also with the survey hosting vendor and with Spout’s partners responsible for the technical implementation of Spout.com.

In addition, we were concerned about how long it might take to recruit participants for both research phases whose characteristics matched Spout’s target audience: people who watch at least three movies a month and who regularly consult online information (such as Netflix, Blockbuster Online, Yahoo Movies) to decide how to invest their movie consumption time.

Qualitative Research Goals and Methodology

Spout’s high-level questions exemplify the integration of marketing and usability research—they are central to the marketing strategy and branding for Spout.com, and they’re rooted in the user experience.

When we planned the qualitative usability testing that began the project, we focused on questions that would be best answered by observing hands-on use.

To explore these issues, we asked the usability test participants to perform four tasks and we observed what site features they used and why. We asked participants to find out about a movie which we knew the site provided rich content for, and then to look for a movie we knew had no member-contributed reviews. Finally, we asked participants to write a review on the last movie they saw and noted their reactions to the review creation process.

Qualitative Research Results and What Spout Did

The qualitative usability test yielded many positive findings for marketing. The majority of the participants successfully used the reviews on Spout.com to help them decide on a movie, even when (in the first task) they had said they usually don’t do so. Although participants didn’t immediately understand the community aspects of the site, after exploring the features, most found them interesting and intriguing.

Two interrelated areas confused the usability test participants: they were uncertain exactly what Spout was and how to use the site to the best advantage. Participants wondered whether the primary goal of the site was to:

Participants also wanted to know why they should sign up—how would they benefit, and what disadvantages might there be (such as receiving too many advertising emails).

The second area of confusion was the terminology on the site. The terms used for different features (tags, talk, details, experts, people) were either ambiguous to participants or meant something different from what they expected. For example, participants were not sure who the experts were or how they became “experts.”

Spout immediately began addressing the two problem areas the qualitative usability testing uncovered. They added choices on the home page to help visitors see benefits. Clicking on “Learn more” (Figure 1) took visitors to a one-minute tour (Figure 2), which we then tested in the quantitative research.

Screen images of the Spout.com landing page
Figure 1: Spout.com landing page

Screen image from the Spout.com tour
Figure 2: A Page of the tour

Quantitative Research: Testing First Impressions

When we worked with Spout stakeholders to identify the objectives for the quantitative research, we learned that Spout wanted to focus on initial user reactions to a new site landing page, an updated Spout tour, and a new MovieMind tool (Figure 3) that helped people select movies.

In our initial design of the task-based survey, participants moved back and forth between Spout.com and an online survey site run by a third-party vendor. Participants started with background questions in the survey, switched to Spout.com to perform each task, then returning to the survey to answer questions about the task. However, during review we found snags in this approach: problems arose with synchronization of data between the sites, and the back-and-forth process made the survey take too long to complete.

Fortunately, users could experience the landing page and the tour with static screens, so we were able to simplify the research design by illustrating the survey questions with screen captures, plus just one branch to the live Spout.com site for trying the MovieMind tool. The survey questions addressed participants’ specific reactions to the screens. Taking the tour and using MovieMind gave participants a realistic and immediate context for the opinions we elicited about usefulness, appeal, and value of the site.

Screen image of MovieMind
Figure 3: Opening Page for the MovieMind Tool

We also encountered some of the anticipated problems with participant screening. With our original criteria, the survey vendor would have to send out more than 132,000 invitations to receive 350 completed surveys from qualified participants, which required much more than the available budget. We compromised by adjusting the screening criteria (accepting participants who visit only one rather than two movie sites) and by reducing the number of participants to 250.

Results from the Quantitative Research

After analyzing the survey results, we concluded that:

Although the tour did not get rave reviews, it succeeded in describing Spout to the majority of new users. Also, after going through the tour screens, more of the participants said they would become Spout members. When we asked participants their opinions of the recommendations the MovieMind provided, half said the MovieMind recommendations were appropriate or very appropriate.

Overall, the quantitative research confirmed that Spout had addressed the usability test participants’ confusion about the purpose of the site. It also validated Spout’s approach to encouraging first-time visitors to join the community.

Integrated Qualitative and Quantitative Research for Spout and Others

Spout’s program of integrated qualitative and quantitative research was unusual because most startups don’t yet understand the value of user research, and fewer still will plan longitudinal research programs. All of Tec‑Ed’s other integrated qualitative and quantitative research projects have been for Fortune 500 companies in business over ten years with large in-house user experience groups.

Another memorable aspect of the project was Spout’s willingness to collaborate on innovative approaches such as our task-based survey methodology, which required cooperation from the IT specialists maintaining and enhancing Spout.com. Again, many startups would not invest scarce technical resources to learn more about their customers.

On the other hand, the Spout project was similar in many ways to other integrated qualitative and quantitative research projects we have performed recently. Some of the issues we consider for all such projects include:

Our Spout project was especially rewarding because the many stakeholders at Spout, BBK Studios, and Tec-Ed worked effectively together in a successful partnership. This case history illustrates how you don’t have to work for a Fortune 500 company to carry out cutting-edge user experience research. Smaller organizations and startups can also benefit from conducting integrated qualitative and quantitative research.

Stephanie Rosenbaum is CEO of Tec-Ed, Inc., a fifteen-person user experience consultancy with offices in Michigan, California, and New York; clients include eBay, Google, Microsoft, and Yahoo! She co-authored a chapter in (ital)Cost-Justifying Usability, An Update for the Internet Age, and contributed an invited chapter on “The Future of Usability Evaluation" to (ital)Maturing Usability (Springer, 2008). A charter member of UPA, Stephanie has attended every UPA conference and has presented at most of them.


UPA logo
Usability Professionals' Association

promoting usability concepts and techniques worldwide

User Experience Magazine is by and about usability professionals, featuring significant and unique articles dealing with the broad field of usability and the user experience.

This article was originally printed in User Experience Magazine, Volume 7, Issue 2, 2008.

© Usability Professionals' Association
Contact UPA at http://www.usabilityprofessionals.org/about_upa/contact_upa.html