Practical Statistical Techniques for the Usability Practitioner
If you dont measure it you cant manage it: Is Product A faster than Product B? Did we meet our goal of a 75% completion rate? Learn how to conduct and interpret appropriate statistical tests on usability data then communicate your results in easy to understand terms to stakeholders.
Half Day Tutorial by
Jeff Sauro (Oracle)
- Advanced Practitioners (AP)
6:30pm to 9:30pm on Monday, June 08, 2009
About the Half Day Tutorial
- INTRO: The tutorial will begin with briefly reviewing the most common ways to measure usability at the task-level: completion rates, task time, errors and task-level satisfaction. (15 Min)
- Big-4 Measures are operationalized metrics of ISO 9241-pt 11.
- Understanding the role of chance in sampling data: intuition can be helpful and harmful
- Objective: Clearly understand both the limits and data available from small sample usability data through use of confidence intervals (30 Min)
- Confidence Intervals
- Interval estimation and point estimation
- Confidence Interval around a Completion Rate (Adjusted Wald)
- Compute confidence intervals using excel and using the online calculator using instructor or participant provided data for a task completion rate(5 Min)
- Confidence Interval around mean task time (t-statistic)
- Dealing with Non-Normal Time data
- The importance of graphing time-data
- Mean, Median or Geometric Mean?
- Compute confidence intervals using excel and using the online calculator using instructor or participant provided data for task times (5 Min)
- What about Sample Size?
- Objective: Determine if a usability test task has met or exceeded a goal (e.g. users can complete the transaction is less than 2 minutes). (30 Min)
- Identifying Specification Limits for task time and completion rates
- Comparing task completion rate against a benchmark
- e.g. 9/10 completed the task does this meet or exceed our goal of 75% completion rate for 1st time use?
- 1-proportion test using exact binomial probabilities
- Compare a task completion rate against a stated benchmark (e.g. 75%) using excel and using the online calculator with instructor or participant provided data (5 Min)
- Comparing Avg. Task Time against a specification limit
- e.g. The average time from 12 users was 2 minutes, does this meet or exceed our goal that users can complete the task in less than 2.5 minutes?
- 1-sample t test
- Compare the average task time against a stated specification (e.g. 2 minutes) using excel and using the online calculator with instructor or participant provided data (5 Min)
- What about Sample Size?
- Objective: Be able to compare two interfaces or versions (A/B Testing) by showing statistical significance (e.g. Product A takes 20% less time to complete a task than Product B p