upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

When 100% Really Isn’t 100%: Improving the Accuracy of Small-Sample Estimates of Completion Rates

James R. Lewis and Jeff Sauro

Journal of Usability Studies, Issue 3, Volume 1, May 2006, pp. 136-150

Abstract

Small sample sizes are a fact of life for most usability practitioners. This can lead to serious measurement problems, especially when making binary measurements such as successful task completion rates (p). The computation of confidence intervals helps by establishing the likely boundaries of measurement, but there is still a question of how to compute the best point estimate, especially for extreme outcomes. In this paper, we report the results of investigations of the accuracy of different estimation methods for two hypothetical distributions and one empirical distribution of p. If a practitioner has no expectation about the value of p, then the Laplace method ((x+1)/(n+2)) is the best estimator. If practitioners are reasonably sure that p will range between .5 and 1.0, then they should use the Wilson method if the observed value of p is less than .5, Laplace when p is greater than .9, and maximum likelihood (x/n) otherwise.

Practitioner's Take Away

Download Article

When 100% Really Isn't 100%: Improving the Accuracy of Small-Sample Estimates of Completion Rates