upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Unexpected Complexity in a Traditional Usability Study

Tharon W. Howard

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205

Article Contents


The procedure used in this study was a traditional think-aloud protocol analysis. There were three major parts to the study:

  1. Pre-test interview: where demographic and background data about users were collected.
  2. Think-aloud protocols: where users were presented with scenarios requiring that they complete tasks and say aloud what they are thinking and doing as they perform the tasks.
  3. Post-test interview: where users were asked to reflect critically on their experiences and to compare the texts examined.

Pilot testing was conducted on the instruments to ensure that the questions, instructions, and scenarios were understandable and to verify that the data collection instruments functioned properly, but data from the pilot testing are not used in this article.

Pre-test interview and subject profiles

After going through the informed consent statement agreement and giving their permission to be videotaped, participants in the study were asked to participate in a pre-scenario interview. This interview collected basic demographic information about participants' experience with high school English classrooms and helped us gauge whether or not the participants were representative of typical freshman composition students.

Table 1 provides a breakdown of the 12 participants who were recruited from 6 different composition classes and were paid $75 for their participation. Because grammar handbooks are used in both 2-year and 4-year colleges, 6 of the participants (4 males and 2 females) were from a 2-year community college, and 6 of the participants (3 males and 3 females) were from a 4-year university. All of the participants were either 18 or 19, all 12 were in their first semester of college, and most importantly, all 12 participants were currently enrolled in their first college-level composition course.

Table 1. Personal Background Information
User # Age Gender Race Major High School GPA 4.0 Scale
1 18 M A Industrial Engineering 4.2/5.0 3.36
2 18 F C Communication 4.8/5.0 3.84
3 19 M C Undeclared 4.02/5.0 3.22
4 18 F C Spanish & International Trade 4.24/5.0 3.4
5 18 M A Computer Science 3.7/4.0 3.7
6 18 F A Chemistry 3.5/4.0 3.5
7 19 M C Univ. Transfer-Electrical Engineering 3.84/4.0 3.84
8 19 M C Univ. Transfer-Electrical Engineering 3.6/4.0 3.6
9 19 M C Undeclared 3.4/4.0 3.4
10 18 M C Univ. Transfer-Business Management 2.8/4.0 2.8
11 18 F C Univ. Transfer-Business Management 3.2/4.0 3.2
12 18 F C Univ. Transfer-General Studies (wants to go to Medical School) 3.2/4.0 3.2
TOTALS 18.3         3.42

A=African American

We also collected data about grades, SAT scores, majors, and other information in order to show that users were fairly representative of freshman composition users in the Southeast.

Previous | Next