Getting Your Money Back: The ROI of remote unmoderated user research

Remote unmoderated user experience research is a valu­able methodology to include in any consultant’s toolkit. The ability to conduct task-based research that collects both qualitative and quantitative data enhances return on investment (ROI) significantly. This approach can also be used to augment other research methodologies.

This methodology synthesizes clickstream data with attitudinal and task success data, all taken from the same people at the same time as they interact with a website. Data regarding user intent are correlated with information regarding how they go about achieving their goals, and why the experi­ence does or does not work well. Not only does this methodology provide an increased ROI for the cost of conducting a study, but customer experience researchers can use it to investigate user behaviors at every stage of the website life cycle.

Target website with survey ask bubble

Targeting users with intercepts.

An Effective Research Method

Intercepting visitors to a website and inviting them to participate in a remote usability study is arguably the most effective way to understand visitors’ goals and intentions. Many companies assume that they know why people come to their site, but when they ask real visitors, the responses may come as a surprise. Other companies don’t understand why people came to their site and build their site solely from a busi­ness perspective.

A large sample size gives business stakehold­ers more confidence in the results of research studies focusing on the overall health and success of a website or prototype design. With increased confidence in the research data, the results can be used to inform the design process by helping website development teams prioritize changes that need to be made to the site design.

Remote unmoderated user research is an effec­tive method providing many benefits (Table 1).

Research Benefit Details
Increase geographic reach
  • Recruit participants from around the country and world
  • Evaluate multiple sites simultaneously: multiple country sites, competitor sites or brand sites such as the Hilton and Starwood Hotels
Provide a local language experience for participants during the study
  • Present system messages, buttons, and other UI elements in local languages • Translate task scenarios and questions •  Evaluate localized websites
Eliminate moderator bias
  • Manage the participant flow and questioning with conditional logic, rather than moderator influence and intervention
Provide a natural environment
  • Participants use their own machines at a time that is convenient for them, and experiencing their own internet connection speeds
Acquire research data in a timely manner
  • Evaluate multiple sites or locations simultaneously • Test prototypes and iterate quickly
    Scrum/agile development teams use this method because researchers can obtain feedback from a sample of participants, pause the study, obtain insights, iterate the designs, resume the study, and repeat the process until the team is satisfied with the design.
Collect large samples
  • Samples sizes of 200-400 provide statistical significance
  • Screening participants so they meet the profile and set quota requirements and study perform­ance criteria-such as minimum task browse time and minimum number of characters for the freeform responses—to increase confidence intervals
Obtain comparable results to in-person studies
  • Synthesize behavioral with attitudinal information, allowing researchers to uncover the why behind someone’s behavior
  • Directed tasks provide context for the behavior to help understand the why behind behavior
Obtain data from actual site visitors performing their real tasks while they perform them
  • Understand the true intent and experience of real site visitors
  • Learn why people visit a site, who they are, where they go, what they do, whether or not they are successful, satisfied, and much more.

 

A Cost Efficient Research Method

Remote user research is a very efficient way to conduct research.

  • Multiple sites and countries can be evaluated, simultaneously reducing cost and time commitments.
  • Both attitudinal and behavioral data are captured on significantly large sample sizes as compared to more traditional research.
  • Multiple research projects can be conducted simultaneously by one researcher versus needing a team of researchers.

The cost benefits for remote unmoderated user research are listed in Table 2.

Cost Benefit Details
Reduced costs
  • Teams of researchers do not have to travel to locations around the country or world.
  • The remote methodology eliminates lab facility costs and the cost of test computers used in a market research facility.
Fewer difficulties conducting the research
  • Eliminates the need to schedule times for participants to complete research
  • Eliminates the time and effort of travel or organizing local research teams
Easier (and faster) data preparation
  • Automatic capture of performance measures: time on task, time on page, page load time, navi­gation paths, and client-side actions (hover-overs, text change, clicks, and scrolling)
  • Participants type in their own responses to the questions or comments
  • The research tool aggregates the data and calculates correlations automatically
  • Individual and aggregated clickstream paths are prepared automatically
  • Data can be downloaded to a variety of file formats, such as PowerPoint, Word, Excel, SAS, SPSS

 

Research Results Provide Real Insights

Understanding users’ goals, attitudes, and behavior provides real insight into why they navigate in a particular way. By look­ing at the behavior of the users through the clickstream data, you gain insights into their attitudes. These insights lead directly to improvements in the design of the site.

For example, a drop-off analysis was conducted on a booking process funnel. While the percentage of people that drop­off at each stage in the process can also be obtained by web analytics tools, the designer needs to understand why the users are dissatisfied or having problems. Remote, task-based studies tell you why they drop off—you can see exactly which people drop off, and at the same time know why they drop off. As a result, you have greater confidence in the reliability and accuracy of the data.

World of Warcraft page with survey ask

Intercepts can match the style of the website.

Remote, unmoderated research is particularly effective for conducting “true intent” and “open” web research. Attitudinal and behav­ioral data are gathered from people naturally visiting a website for true intent research, just as market researchers outside a store can. This method helps us understand why visitors come to a website, whether or not they are successful and satisfied, what they actually do on the site, and what impact the site experience has on the brand perception. With large sample sizes of 400 or more, it is feasible to filter the data by visit intent.

In open web research, users must be able to freely and naturally explore the Internet. Researchers learn which sites are “top-of-mind,” which are most popular and why, how the research is approached, and how specific websites are found. Remote, unmoderated research provides the ability to trigger ques­tions when users move from one site to another, providing deeper insights into users’ motivations and behavior.

Patterns in user behavior and sites visit­ed emerge with large sample sizes of 400 or more in open web research. These pat­terns help researchers understand how users find specific sites (if at all), how the web can be used to develop awareness and preference for specific products and servic­es, and whether or not users naturally navigate to specific sites (If so, how and why? If not, why not?)

Companies use remote unmoderated research to conduct a variety of research, such as:

  • BRAND AND VALUE PROPOSITION-Are we promoting the brand and value proposition? Do visitors know what we do and who we are?
  • PROTOTYPE EVALUATION -Will the next site be successful? Will users find it compelling and helpful?
  • TRUE INTENT – Why do users come to the site, and how well does it deliver on those expectations?
  • USABILITY – What are the strengths, weaknesses, and opportunities to improve satisfaction and conversion on my site?
  • COMPETITIVE RESEARCH-How do we compare to the competition? What strategies will help us win?
  • INTERNATIONAL STUDIES -How are we performing in key markets internationally?
  • NAVIGATION AND ARCHITECTURE-How well does my site enable key task accomplishment?
  • OPEN WEB RESEARCH-How do users find our site? How can we use the web to develop awareness and preferences?
  • CARD SORTING STUDIES -What are users’ expectations of the information archi tecture? What category names should be
    used for navigation?
  • PERSONA DEVELOPMENT -Who is visiting our site? What psychographics best match our target users?

Remote, unmoderated, task-based, qualita­tive, quantitative user research is an excellent addition to any consultant’s toolkit. It does not replace other methods, but with a larger sam­ple size, geographic reach, globalization, and ability to quickly run studies it adds value on it own merits

Farnsworth, C. (2008). Getting Your Money Back: The ROI of remote unmoderated user research. User Experience Magazine, 7(3).
Retrieved from http://www.uxpamagazine.org/getting_your_money_back/

Comments are closed.