upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Heuristic Evaluation Quality Score (HEQS): Defining Heuristic Expertise

Shazeeye Kirmani

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 31-48

Article Contents


A world wide heuristic evaluation competition that was part of Usability Professionals’ Association’s (UPA) World Usability Day was hosted by UPA, Bangalore online in November 2007. The competition details were as follows:

Contestants had to sign an honor statement stating that they would not take more than an hour to complete the evaluation and that the evaluation was the sole effort of the contestant. The competition lasted for 2 weeks and entries were submitted in a particular format (see Table 1). The UI parameters and severity categories were taken from the initial HEQS paper (Kirmani & Rajasekaran, 2007).

Table 1. The Evaluation Format.
Issue The other 2 steps in the process of getting advice for a particular symptom are barely visible.
UI Parameter Visual Design
Severity Major Issue

Contestants were encouraged to participate via various methods such as sending links to the competition via individual e-mails, blogs, social networking sites, and usability communities. Anyone could participate. They were all directed to the World Usability Day website (see Figure 1) where the event was hosted online. The event page had a downloadable presentation with the demographic data collection form, evaluation format, scope of the evaluation, and judgment criteria. Each contestant used their own evaluation criteria to conduct the 1 hour evaluation. They had to e-mail their entries along with the filled demographic form to a given e-mail address.

Figure 1. Competition details and instructions hosted on the World Usability Day website.

Figure 1. Competition details and instructions hosted on the World Usability Day website.

The competition was judged the same way as described in the previous HEQS paper (Kirmani & Rajasekaran, 2007). The judges were given a benchmark of a collation of all the issues (200 plus issues) of all the contestants. The three judges who were heuristic experts followed the process listed below.

  1. Each judge rated each issue of the 200 plus issues as valid or invalid individually.
  2. Each judge categorized the severity and UI parameter of each valid issue individually.
  3. They got together to discuss each issue and its severity and UI parameter categorization.
  4. If they did not agree on the issue validity, severity, or UI parameter categorization they discussed it together to arrive at a final consensus.
  5. This finalized list of valid issues with their appropriate severity and categorization was used to judge each contestant.
  6. Judges were requested to write down their thoughts on the categorization process to later discuss ways to improve it.
  7. Each issue of a contestant’s entry was matched to the finalized list and weights of 5, 3, or 1 were awarded based on the severity of the issue (5 for every showstopper, 3 for every major issue, and 1 for every irritant). Summing up the scores for all the issues one arrived at the HEQS for each contestant.
  8. If contestants incorrectly categorized issues for severity or UI parameter they were re‑categorized to arrive at their scores. Contestants were asked to enter the severity and UI parameter to gain insight into the categorization process. For example, the way the issue was worded depended to a great extent to the way it was categorized. This qualitative data helped to improve the categorization process.

Demographic Data

Twenty contestants took part in the competition. Table 2 summarizes the demographic data.

Table 2. Demographic Data of the Contestants.
Parameter Average Range
Age 28.4 years 22-34 years
Time spent as heuristic evaluator 23.7 months 0-120 months
Time spent as a usability practitioner 30.7 months 0-144 months
Time spent as a domain expert (i.e. healthcare) 4.2 months 0-24 months
Confident of winning (self rating on a scale of 5 where 5=absolutely win and 1=never win) 4.1 1-5
Gender 6 females and 14 males4
Location 6 states and 2 continents

Karnataka, India
Mumbai, India
Gujarat, India
Chennai, India
New Jersey, US
California, US

Previous | Next