upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Usability Evaluation of a Tag-Based Interface

Rajinesh Ravendran, Ian MacColl, and Michael Docherty

Journal of Usability Studies, Volume 7, Issue 4, August 2012, pp. 143 - 160

Article Contents

Method and Procedure

The following sections discuss the method, evaluation procedures, interfaces, and participants.


To evaluate the usability of the interfaces, we adapted the System Usability Scale (SUS; Brooke, 1996) by replacing system with website. The SUS is a 10-item questionnaire, with Likert scales, that gives an overview of perceived usability in terms of efficiency, effectiveness, and satisfaction as perceived by the user. According to Tullis and Stetson (2004) who compared SUS against other usability questionnaires specifically for website usability assessment, SUS yields among the most reliable results across sample sizes. To yield reasonably reliable results, at least 12-14 participants should be used in a study. Furthermore, as an extension we added an 11th item on “user-friendliness” with adjective descriptions of rating levels of 1-7 (Worst Imaginable, Awful, Poor, OK, Good, Excellent, and Best Imaginable). The purpose of this item was to inquire about the summative experience of participants. This strategy was suggested by a study that empirically evaluated nearly 10 years worth of SUS data collected on numerous products in all phases of the development lifecycle (Bangor, Kortum, & Miller, 2008). The study regarded SUS as a highly robust and versatile tool used in more than 200 studies for usability evaluation. To further ensure SUS is a suitable instrument for our study, we conducted a pilot study with a small group of banking users (see the Procedure - Pilot Study section).

Apart from SUS, we also recorded user activity on the interfaces and logged task completion times. This information was used to examine the actual efficiency and effectiveness, and also to identify potential usability issues on the interfaces


Before the start of the evaluation, each participant filled out a pre-test questionnaire that collected demographic details along with computer, Internet, and online banking experience. Subsequently, each participant was given a set of banking tasks (see Table 2) identified from our pilot study as suitable and generic. The tasks were essentially made up of two key activities: bill payment and fund transfer. Each participant completed the tasks in both the conventional and tag-based interfaces. During testing, we observed each participant and maintained an observation log. After completion of the tasks, each participant received an extended SUS questionnaire, which was then followed by a debriefing session to gather additional feedback.

To ensure the absence of any pre-meditated associations in terms of experience and brand commitment with a participant, a fictional banking website was used called XBANK Online Banking. In order to further engage participants with the study, we provided a scenario that outlined XBANK as a new online banking provider committed to better user satisfaction. Participants were told the goal of the XBANK study was to evaluate the usability of two interfaces. The participants were informed that they were a select group of users who were selected to be part of the evaluation. For the purpose of the evaluation, participants were told that he/she owned five accounts: Savings, Everyday, MasterCard, Visa, and Check. The order in which the interfaces and tasks were introduced was counterbalanced.

The study's goals, objectives, and tasks were explained in an information sheet. The evaluation procedure was detailed at the beginning of each session and in order to familiarize each participant with the interfaces, a brief demo and a test activity was given. Participants assigned tags to banking resources on their own with the tag-based interface.

Table 2. Evaluation Tasks

Table 2

Post-evaluation debriefing

A post-evaluating debriefing was conducted with each participant upon completion of the usability questionnaire. We asked participants about the overall experience with both interfaces, particularly the challenges and issues faced with the tag-based interface. Additionally, we questioned participants about the degree to which information was made available in both interfaces and their preference. We also asked participants on their perception of tag suggestions as a way to ease tagging. Finally, we sought feedback on the design of the tag-based interface and potential improvements.

Pilot study

A pilot study involving a small group of eight banking users was carried out (Ravendan, MacColl, & Docherty, 2012). The goal of the pilot was to tease out early design impediments and other usability issues prevalent in both interfaces, and to debug the test procedure. Also, we asked participants their five most common online and/or mobile banking tasks. The pilot study helped assess the adapted SUS questionnaire and its ability to offer a reflective and consistent view of users’ perceived usability with the online and mobile contexts. The results showed that users’ views were reflective and consistent of their feedback gathered during debriefing. There was a notable difference between the SUS scores for the conventional and tag-based interfaces. The difference was especially apparent with banking users without prior experience in mobile banking.


For the purpose of this evaluation, we developed a tag-based prototype and a test website that mimicked the conventional banking interface in both online and mobile contexts. We designed the conventional interface with reference to the online and mobile banking websites of Commonwealth Bank and Suncorp Bank (mentioned above). The following sections outline the two interfaces that were evaluated in this study.

Conventional interface

The conventional interface was based on standard HTML objects (dropdowns, checkboxes, tables, and menus) with minimal customization (see Figure 6). From our observation of websites with conventional interfaces, only remembrance-based customization is being supported in the conventional interface. This is achieved via dropdown selection where previously entered information is remembered. For instance, when a user selects a biller that he or she has paid to before from the dropdown selection, the biller details are automatically populated, avoiding the need for the user to re-enter the biller details.

Figure 6/p>

Figure 6. Conventional interface

Tag-based interface

We designed an early web-based prototype with tag integration for a few key resources namely bank account, biller, and reference. Tag-based visualization that included tag clouds and individual tags represented as clickable boxes were used. In Ravendran et al. (2011a, 2011b), a detailed discussion on the design and implementation of the tag-based interface and the different customization types in both online and mobile banking contexts was presented. In this study, we evaluated our tag-based prototype against the conventional banking interface.

Figure 7

Figure 7. Tag-based interface (online)


The population of interest for this study is online and mobile banking users. A total of 30 banking users were randomly recruited from the university: 17 males and 13 females between the age group of 21 to 50 (21-30=60%, 31-40=33%, 41-50=7%). The recruitment process was conducted through emails, which were sent to university staff and students. In order to compensate participants for their time and effort, we offered a gift card worth $15. Altogether, there were 22 students and 8 staff who participated in the study. The larger number of participants from the younger age group (21-30) was pertinent to this study given banking customization is most appealing and relevant among this age group (Rahim & JieYing, 2009). According to the pre-test questionnaire, all participants had at least one active online banking account at the time of participation and were familiar with online banking with no less than one year of experience. However, not all participants had prior experience in mobile banking. Figure 6 shows the mobile banking usage and only 54% of the participants had prior experience with mobile banking, while 46% of the participants had never used mobile banking before. Some of key reasons cited for not using mobile banking include security and privacy concerns, high risk of mistyping on a mobile device, and preference for a bigger display screen. Figure 7 shows the most common banking activities carried out by participants via their online/mobile banking provider, which include fund transfer, bill payment, check balance, view history, and credit card payment. Participants had very similar levels of experience with computers and the Web, and indicated that they enjoy using computers and the Internet. Most participants (90%) were familiar with the concept of tags primarily through websites such as Facebook and YouTube, but not in the financial sense. Participation was entirely voluntary, and each individual gave written consent to participate in the study.

Figure 8

Figure 8. Mobile banking usage

Figure 9

Figure 9. Banking activities

Previous | Next