Heuristic Evaluation Quality Score (HEQS): Defining Heuristic Expertise
Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 31-48
Article Contents
Method
A world wide heuristic evaluation competition that was part of Usability Professionals’ Association’s (UPA) World Usability Day was hosted by UPA, Bangalore online in November 2007. The competition details were as follows:
- Nature of the website: A healthcare site where evaluators put in symptoms and the website provides advice.
- Scope of the evaluation: Five scenarios were given to evaluators.
- Find all conditions related to a cough.
- Check all symptoms associated with a cough.
- Edit the symptoms.
- Find related articles.
- E-mail an article to yourself.
- Time to do the evaluation: 1 hour
- Demographic data collected: age, gender, experience, location, and confidence to win
Contestants had to sign an honor statement stating that they would not take more than an hour to complete the evaluation and that the evaluation was the sole effort of the contestant. The competition lasted for 2 weeks and entries were submitted in a particular format (see Table 1). The UI parameters and severity categories were taken from the initial HEQS paper (Kirmani & Rajasekaran, 2007).
| Issue | The other 2 steps in the process of getting advice for a particular symptom are barely visible. |
| UI Parameter | Visual Design |
| Severity | Major Issue |
Contestants were encouraged to participate via various methods such as sending links to the competition via individual e-mails, blogs, social networking sites, and usability communities. Anyone could participate. They were all directed to the World Usability Day website (see Figure 1) where the event was hosted online. The event page had a downloadable presentation with the demographic data collection form, evaluation format, scope of the evaluation, and judgment criteria. Each contestant used their own evaluation criteria to conduct the 1 hour evaluation. They had to e-mail their entries along with the filled demographic form to a given e-mail address.
Figure 1. Competition details and instructions hosted on the World Usability Day website.
The competition was judged the same way as described in the previous HEQS paper (Kirmani & Rajasekaran, 2007). The judges were given a benchmark of a collation of all the issues (200 plus issues) of all the contestants. The three judges who were heuristic experts followed the process listed below.
- Each judge rated each issue of the 200 plus issues as valid or invalid individually.
- Each judge categorized the severity and UI parameter of each valid issue individually.
- They got together to discuss each issue and its severity and UI parameter categorization.
- If they did not agree on the issue validity, severity, or UI parameter categorization they discussed it together to arrive at a final consensus.
- This finalized list of valid issues with their appropriate severity and categorization was used to judge each contestant.
- Judges were requested to write down their thoughts on the categorization process to later discuss ways to improve it.
- Each issue of a contestant’s entry was matched to the finalized list and weights of 5, 3, or 1 were awarded based on the severity of the issue (5 for every showstopper, 3 for every major issue, and 1 for every irritant). Summing up the scores for all the issues one arrived at the HEQS for each contestant.
- If contestants incorrectly categorized issues for severity or UI parameter they were re‑categorized to arrive at their scores. Contestants were asked to enter the severity and UI parameter to gain insight into the categorization process. For example, the way the issue was worded depended to a great extent to the way it was categorized. This qualitative data helped to improve the categorization process.
Demographic Data
Twenty contestants took part in the competition. Table 2 summarizes the demographic data.
| Parameter | Average | Range |
|---|---|---|
| Age | 28.4 years | 22-34 years |
| Time spent as heuristic evaluator | 23.7 months | 0-120 months |
| Time spent as a usability practitioner | 30.7 months | 0-144 months |
| Time spent as a domain expert (i.e. healthcare) | 4.2 months | 0-24 months |
| Confident of winning (self rating on a scale of 5 where 5=absolutely win and 1=never win) | 4.1 | 1-5 |
| Gender | 6 females and 14 males4 | |
| Location | 6 states and 2 continents Karnataka, India Mumbai, India Gujarat, India Chennai, India New Jersey, US California, US |
|