Unexpected Complexity in a Traditional Usability Study
Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 189-205
Article Contents
Methodology
The procedure used in this study was a traditional think-aloud protocol analysis. There were three major parts to the study:
- Pre-test interview: where demographic and background data about users were collected.
- Think-aloud protocols: where users were presented with scenarios requiring that they complete tasks and say aloud what they are thinking and doing as they perform the tasks.
- Post-test interview: where users were asked to reflect critically on their experiences and to compare the texts examined.
Pilot testing was conducted on the instruments to ensure that the questions, instructions, and scenarios were understandable and to verify that the data collection instruments functioned properly, but data from the pilot testing are not used in this article.
Pre-test interview and subject profiles
After going through the informed consent statement agreement and giving their permission to be videotaped, participants in the study were asked to participate in a pre-scenario interview. This interview collected basic demographic information about participants' experience with high school English classrooms and helped us gauge whether or not the participants were representative of typical freshman composition students.
Table 1 provides a breakdown of the 12 participants who were recruited from 6 different composition classes and were paid $75 for their participation. Because grammar handbooks are used in both 2-year and 4-year colleges, 6 of the participants (4 males and 2 females) were from a 2-year community college, and 6 of the participants (3 males and 3 females) were from a 4-year university. All of the participants were either 18 or 19, all 12 were in their first semester of college, and most importantly, all 12 participants were currently enrolled in their first college-level composition course.
| User # | Age | Gender | Race | Major | High School GPA | 4.0 Scale |
|---|---|---|---|---|---|---|
| 1 | 18 | M | A | Industrial Engineering | 4.2/5.0 | 3.36 |
| 2 | 18 | F | C | Communication | 4.8/5.0 | 3.84 |
| 3 | 19 | M | C | Undeclared | 4.02/5.0 | 3.22 |
| 4 | 18 | F | C | Spanish & International Trade | 4.24/5.0 | 3.4 |
| 5 | 18 | M | A | Computer Science | 3.7/4.0 | 3.7 |
| 6 | 18 | F | A | Chemistry | 3.5/4.0 | 3.5 |
| 7 | 19 | M | C | Univ. Transfer-Electrical Engineering | 3.84/4.0 | 3.84 |
| 8 | 19 | M | C | Univ. Transfer-Electrical Engineering | 3.6/4.0 | 3.6 |
| 9 | 19 | M | C | Undeclared | 3.4/4.0 | 3.4 |
| 10 | 18 | M | C | Univ. Transfer-Business Management | 2.8/4.0 | 2.8 |
| 11 | 18 | F | C | Univ. Transfer-Business Management | 3.2/4.0 | 3.2 |
| 12 | 18 | F | C | Univ. Transfer-General Studies (wants to go to Medical School) | 3.2/4.0 | 3.2 |
| TOTALS | 18.3 | 3.42 |
A=African American
C=Caucasian
We also collected data about grades, SAT scores, majors, and other information in order to show that users were fairly representative of freshman composition users in the Southeast.