Half Day Tutorial Details

Download: Schedule of Events (PDF) subject to change
By Day - Selected
By Presenter
By Track
View My Agenda

Unmoderated Usability Testing: How to create, run, and analyze your own study

Introduction for practitioners that want to learn more about unmoderated remote usability testing

This tutorial will provide all the information you need to decide if unmoderated testing is the right fit for you and, if so, how to create, run, and analyze your own study. Exercises will also be incorporated to give you experience working with unmoderated tools and data from studies.

Half Day Tutorial by Kyle Soucy (Usable Interface), Tom Tullis (Fidelity Investments)
User Research
9:00am to 12:30pm on Friday, June 08, 2012

About the Half Day Tutorial

There has been a surge in the number of tools that are available for conducting unmoderated remote usability testing—and this surge is changing the usability industry. As usability professionals, it's important to know what these tools are capable of and how to work with them in order to make an informed decision about incorporating them into your usability toolbox.

This tutorial will provide an introduction to unmoderated usability testing and it will cover everything from how to set up your own study, to analyzing the data. You will walk out of this tutorial with all the information you need to decide if unmoderated testing is the right fit for your needs and, if so, how to create, run, and analyze your own study. You will have the opportunity to take part in an actual unmoderated usability study and get some hands-on experience working with the data from it. Since we believe that task creation is one of the most critical components of a successful unmoderated study, you will also have the opportunity to practice writing your own tasks. An excellent hand out will be given out listing all the available unmoderated usability testing tools on the market with descriptions and pricing information — truly a great resource to have at your fingertips!

To clarify, there are a lot of tools on the market that claim to be “usability testing tools”, but don’t actually have the capability of doing usability testing with participants through task elicitation. This tutorial will be focused on unmoderated usability testing tools that actually simulate traditional usability testing by asking participants to complete a series of tasks using a user interface and answering questions about their experience.


Introduction to Unmoderated Usability Testing

  • What is it?
  • What you can learn
    • Quantitative vs. qualitative data
  • How actionable is the data?
  • Benefits & drawbacks
  • When to conduct unmoderated testing?
    • Moderated vs. unmoderated
  • Example study: MeasuringUX.com

Usability Study Design

  • Comparing alternative designs
    • Between subjects
    • Within subjects
  • Comparison to baseline or previous study
  • Competitive usability study
  • Recruiting options & honorariums
  • Task creation
    • Exercise - Writing your own tasks
  • Screening questions
    • Identifying and removing "Speeders & Cheaters"
    • Identifying and handling outliers
  • Dry run & Launching the study
  • Setting an unmoderated usability study up using an online survey tool
  • Exporting results for analysis in Excel

Analyzing Data from Unmoderated Usability Studies

  • Task performance data
    • Task success
    • Task times
    • Efficiency
  • Self-reported data
    • Task (and other) ratings
    • System Usability Scale (SUS)
    • Net Promoter Score (NPS)
    • Product reaction cards
  • Clickstream data
    • Ways of visualizing / analyzing
    • Efficiency: # of clicks beyond minimum
  • Verbatim comments
    • Word clouds
    • Analysis in Excel
    • Specialized text-analysis tools
  • Segmentation Analysis
    • Segmenting by tasks
    • Segmenting by participants
  • Identifying usability issues
    • Using task performance data
    • Using errors
    • Using ratings
    • Using verbatim comments
  • Exercises (at the appropriate points during the tutorial):
    • Calculating and graphing confidence intervals in Excel
    • Analysis of the data from the sample study they did before-hand

Overview of unmoderated testing tools on the market

  • Automated tools with recruiting panels
  • Automated tools with audio and video
  • Automated tools with webcam eye tracking