upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

How Low Can You Go? Is the System Usability Scale Range Restricted?

Philip Kortum and Claudia Ziegler Acemyan

Journal of Usability Studies, Volume 9, Issue 1, November 2013, pp. 14 - 24

Article Contents


Results

System Usability Scale scores for each of the ballots are shown in Figure 4. As can be seen, scores ranged from 15.4 to 93.0. Standard deviations are within the ranges described by Bangor, Kortum, and Miller (2008). The SUS scores are reliably different, F(13) = 50.2, MSE = 299.2, p < .001, η2 = .62. Nine of the 14 group means are below 50, the midpoint of the available SUS range. Figure 5 shows the results for the post-hoc pairwise comparisons (FDR corrected) of all the ballots’ means.

A manipulation check was conducted to insure that ballot presentation order did not impact usability judgments by raters. Specifically, we individually looked at the highest rated ballot overall and the lowest rated ballot overall and compared their mean SUS scores when they were presented in the first five positions and when they were presented in the last five positions. A t-test revealed that there was no significant difference in either case (most usable: t(25)=1.57, p=.130, least usable: t(20)=0.57, p=.576), demonstrating that the order of presentation did not impact SUS scores.

Figure 1

Figure 1. The Modified System Usability Scale used in the study

Figure 2

Figure 2. The eight bottom ranked ballots used in the study (SUS < 40)

Figure 3

Figure 3. The six top ranked ballots used in the study (SUS > 40)

Figure 4

Figure 4. Mean SUS scores for each of the ballots with error bars representing the 95% CI

Figure 5

Figure 5. Post-hoc pairwise comparisons of all the ballots with check marks indicating a difference between ballots’ mean scores

The ballots indicated to be least and most usable are shown in Figure 6. As can be seen, there was high agreement about which of the ballots was most usable, but slightly more disagreement on which ballot was least usable. One ballot (ballot B) was ranked as most usable by one participant and least usable by a few other participants. Both results align with the numeric scores obtained from the SUS.

Figure 6

Figure 6. Counts for the self-reported most usable and least usable ballots

Figure 7

Previous | Next