upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Usability Evaluation of a Tag-Based Interface

Rajinesh Ravendran, Ian MacColl, and Michael Docherty

Journal of Usability Studies, Volume 7, Issue 4, August 2012, pp. 143 - 160

Article Contents


Results

Firstly, we focused on the subjective measures of usability. The following figures 10 and 11 show the computed average SUS scores and summative experience ratings by banking context. The SUS score range was 0-100, and the summative experience rating range was 1-7 (1 = Worst Imaginable, 2 = Awful, 3 = Poor, 4 = OK, 5 = Good, 6 = Excellent, 7 = Best Imaginable).

Figure 10

Figure 10. Average SUS scores (online and mobile)

Figure 11

Figure 11. Average summative experience ratings (online and mobile)

Figures 10 and 11 show higher SUS scores and ratings for the tag-based interface compared to the conventional interface in both online and mobile banking. The average SUS scores for the tag-based interface are 81.1 and 82.8, and the average summative experience rating is 5.7 for online and 5.9 for mobile (both in the Excellent range). In contrast, the average SUS scores for the conventional interface are 64.6 and 64.8, and the average summative experience rating is 4.2 (both in the OK range) for both contexts.

To test the significance of the result, we conducted a paired t-test analysis with an alpha of 0.05 (CI: 95%). The t-test analysis showed the differences in the scores and ratings were significant (n=30, p<0.05) in both online (p<0.002) and mobile (p<0.001) contexts. Given that not all participants of our study had prior experience in mobile banking, we grouped the SUS scores and summative experience ratings by participants’ familiarity with mobile banking. As mentioned above, 54% of participants were familiar with mobile banking. The various levels of familiarity among participants were grouped into two basic categories: inexperienced and experienced.

Figures 12 and 13 show the average SUS scores and summative experience ratings by familiarity with mobile banking.

Figure 12

Figure 12. Average SUS Scores by familiarity with mobile banking

Figure 13

Figure 13. Average summative experience ratings by familiarity with mobile banking

Figure 12 shows the average SUS scores for participants without prior experience are 58.2 and 82.7 for the conventional and tag-based interfaces, respectively. For participants with experience, the average SUS scores are 69.8 and 82.9, respectively. This indicates that participants without prior experience in mobile banking experienced the biggest difference of 24.5%, while the experienced participants recorded a difference of 13.1%. Figure 13 lends support to this outcome by illustrating an increase of 2 rating points among inexperienced participants with an overall individual rating of 4.0 and 6.0 for the conventional and tag-based interfaces. Those ratings are about one rating higher compared to the increase observed with experienced participants with an overall individual rating of 4.4 and 5.8.

Task Completion

All participants managed to complete the given tasks in both contexts. Thus, there was no difference in effectiveness between the two interfaces. The following figure shows the average time spent in seconds on each task by all participants in the online and mobile contexts.

Figure 14

Figure 14. Average task completion times

Figure 14 illustrates that participants in general completed their tasks within a shorter period of time in the online context compared to the mobile context. Overall, the tag-based interface was faster online for 4 out of 6 tasks and 3 out of 6 tasks in the mobile context. The average completion times for the conventional and tag-based interfaces for the online context are 47.4s and 46.7s, and 53.3s and 56.2s for the mobile context. In the online context, participants were faster on the tag-based interface for tasks 2, 4, 5, and 6, but slower for tasks 1 and 3. While in the mobile context, participants were faster on the tag-based interface for tasks 2, 3, and 4 but slower for tasks 1, 5, and 6. Participants spent the most amount of time on task 1 on the tag-based interface in both contexts (60.4s and 69.2s, online and mobile respectively) and a considerably lesser amount of time for the remaining tasks. To ascertain the significance of the task completion times, we conducted a paired t-test analysis with an alpha of 0.05 (CI: 95%). The results showed that the differences of the average task completion times are not significant for either online (p=0.61) or mobile (p=0.36) contexts. Thus, there was no difference in efficiency between the two interfaces.

 

Previous | Next