The U P A voice
June 2005 Contents

Featured UPA Links:

Job Bank

Member Benefits:
Discounts and Special Offers:

Event Discounts

 

amazon.com
Order your usability books now from the Amazon.com link from this site. Part of all sales help fund UPA.

Take Breaks! A Simple Way to Improve Your Heuristic Evaluation Results

by Laura Faulkner
Applied Research Laboratories
The University of Texas at Austin

As primary tools in the usability field, heuristic or expert evaluations can be rich areas for methods studies and improvement. Early results of one methods study suggest that performing evaluations in limited segments, with breaks between each segment, may increase the effectiveness of the evaluator in identifying usability problems.

The basic heuristics derived and published by Rolf Molich and Jakob Nielsen in 1990 (1) have been replicated in many places and forms, with many practitioners having the original set of ten committed to memory. In 1998 Masaaki Kurosu presented research supporting an alternate approach which he and his team termed the Structured Heuristic Evaluation Method (sHEM) (2). In that approach, the research team inserted two significant changes to the evaluation process by: 1) creating a detailed list of heuristics, many with culturally Japanese connotations; and 2) having their research participants perform the evaluation by subject-related sections, breaking the evaluation into several segments with breaks in between. In that study, the participants using the sHEM method found more usability problems than those using the traditional method.

One of the ideas behind breaking the evaluation into segments is to overcome the natural fall-off in performance of the practitioner as the evaluation session progresses. By focusing each segment on specific topics, it can reduce the number of criteria a practitioner is attempting to keep in mind during that portion of the evaluation. Even more important, by taking breaks, the potential mental fatigue of the evaluator is interrupted, allowing the individual to “start fresh” at the beginning of each session, almost as if beginning an entirely new evaluation.

Early results of a study being performed by the Applied Research Laboratories, The University of Texas at Austin (ARL:UT), indicate that simply taking breaks may be the most significant contributor to improved performance of the usability evaluator (3). The study took apart the two aspects of the sHEM study approach, looking at performance using variations that introduce structure alone (smaller sets of criteria coupled with taking breaks), detailed heuristics alone, and the two in combination. With two-thirds of the data in, the clear front-runner so far for number of known usability problems found (4) is the one in which the evaluators simply took breaks, otherwise using their regular approach to a heuristic evaluation.

The moral of this story? You may find yourself more effective in performing heuristic or expert evaluations if you divide them into sections and take breaks between to refresh your thinking. The approach used in the studies was to divide the heuristics themselves into sets. Using those smaller sets of criteria per session, work for an hour, then get away for 30 minutes or more, whether on a complete break or simply to another type of task. You may find that each time you begin it is with the freshness of mind of a whole new evaluation.

Author’s note: If you were to apply the “taking breaks” technique in your next evaluation, our lab would be interested in hearing your experience of it. Please write to laura@arlut.utexas.edu.


(1) Nielsen, J., and Molich, R. (1990). Heuristic evaluation of user interfaces, Proceedings, ACM CHI'90 Conf. (Seattle, WA, 1-5 April), 249-256.

(2) Kurosu, M., Sugizaki, M., Matsuura, S. (1998). Structured Heuristic Evaluation Method (sHEM). Proceedings, Usability Professionals Association Annual Meeting, 3-5.

(3) Faulkner, L. (2005). Structured Software Usability Evaluation: An Experiment in Evaluation Design. Unpublished doctoral dissertation (in progress).

(4) To control for the potential problem of “false positives” that other studies have shown to arise in heuristic evaluation, the ARL:UT study is being conducted on a bogus website intentionally designed with known sets of obvious, mid-range, and subtle usability problems.

  Usability Professionals' Association
140 N. Bloomingdale Road
Bloomingdale, IL 60108-1017
Tel: +1.630.980.4997
Fax: +1.630.351.8490
UPA: office@upassoc.org
Contact the Voice