upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Methodology for Measuring Usability Evaluation Skills Using the Constructivist
Theory and the Second Life Virtual World

Debra J. Slone

Journal of Usability Studies, Volume 4, Issue 4, May 2009, pp. 178-188

Article Contents


Methodology

The following sections discuss the overview, participants, the software application, assessment tools, parameters, scope, knowledge, standardized time, benchmarks, and skills assessment.

Overview

The methods used in the study assessed the heuristic evaluation skills and deep knowledge of usability issues held by graduate students in a Human-Computer interaction (HCI) distance learning course. Table 1 presents an overview of the methodology. The approach is similar to Kirmani and Rajasekaran’s (2007) method of quantifying heuristic evaluation skills.

Table 1. Overview of Assessment Methodology
Method Application

Participants

9 graduate students

Application

Second Life virtual world

Assessment tools

Diaries, checklists, reports, and online discussions

Parameters

Twelve usability guidelines from the Xerox usability checklist (form) and Nielsen and Molich (1990) were used for determining Website usability.

Scope

User interface designed by Second Life developers.

Knowledge

Online lectures, readings, assignments, and online discussions

Standardized time

August 28th through December 8th 2007, for all evaluators

Benchmark

Issues identified by students were collated, repeated issues eliminated, and explanations of violations counted to arrive at a benchmark.

Assessment

Individual ratings were calculated as follows:
Naming of issues = 10 pts each
Identification of other valid issues = 5 pts each
Explanation of identified issues = 10 pts each

Participants

Graduate students in an HCI course at the University of South Florida, School of Library and Information Science spent a semester (fall of 2007) becoming active participants in the virtual community Second Life. Of the 14 students in the course, 9 consented to participation in the study, completing their diaries and other required tasks. Of these, 7 were new to Second Life, while the remaining 2 had used it sporadically. With the exception of 2 single incidences of work use, most interaction with Second Life was from home. Students were reassured that their identities (including that of their avatars, or online persona) would be kept confidential.

The Software Application

Second Life is a virtual community that attempts to recreate an idealized version of the real world (Jones, 2006; Ondrejka, 2004; Rheingold, 2000). In virtual communities, people “become authors not only of text but of themselves, constructing new selves through social interaction” (Turkle, 1995, p.12). For the current study, Second Life provided a platform for teaching the heuristic evaluation method. Second Life was selected for this study because it is well-established and usability problems were not expected to be so great that they overwhelm students. Yet, there was room for improvement with the software.

Assessment Tools

The assessment tools consisted of diaries, online discussions, usability checklists, and usability reports. As part of an assignment, students were asked to keep semester-long diaries of their experiences, feelings, usability problems, progress, and opinions related to Second Life. To encourage the development of the diaries, they were advised to write often, develop a comfortable style of doing so, and visit Second Life on a regular basis. Rather than impose too much structure, the instructor asked students to include simple day and date headings for their diary entries. The purpose for this informal approach was to gather information that the instructor might not have anticipated and to solicit more free-flowing, reflective narrative than might result from a form with checkboxes.

An online discussion forum in Blackboard helped students hone their abilities to identify usability issues. In the forum, they were sometimes able to identify issues they thought were usability problems as personal preference or other problems or system concerns unrelated to Second Life. Students presented and discussed numerous problems based on feedback from the instructor and other students.

At the end of the semester, students conducted heuristic evaluations of Second Life using a usability checklist of established HCI principles. By this time, they had a greater understanding of the Second Life environment and could evaluate specific elements from a user point of view. The checklist allowed students to record usability problems and insights in the framework of a set of 12 usability heuristics.

Students reported the results of their individual evaluation in an independent usability report at the end of the semester. Though typically a report is a joint effort by a set of evaluators, the independent report served the purpose of identifying which student knew what. As part of the report, students identified and attempted to explain specific heuristics violations. The reports included supporting literature where necessary.

The assessment tools ensured a balance between unstructured diaries and discussions and the focus provided by the checklists and reports. Combined, the methods support active involvement of students in the learning process and deep understanding of topics. The instructor reviewed the diaries, usability checklists, and reports to assess student evaluative strengths and weaknesses.

Parameters

The parameters of the usability test covered a detailed set of 12 heuristics from a modified version of the Xerox (1995) usability checklist form. The first ten principles were derived from Nielsen and Molich (1990). This standardized format for rating and evaluation of the system was made available to all students via Blackboard on August 28th, 2007.

The checklist contains a set of questions under the following headings: (a) visibility of system status, (b) match between the system and the real world, (c) user control and freedom, (d) consistency and standards, (e) error handling, (f) recognition rather than recall, (g) flexibility, (h) privacy, (i) minimalist design, (j) help and documentation, (k) skills, and (l) pleasurable and respectful interaction with the user. Each numbered heuristic included a related set of questions designed to expand on that particular problem.

Scope

Students evaluated the Second Life interface. However, they did not evaluate personal or business sites or other landscapes not designed by Second Life developers.

Knowledge

Through online lectures, readings, assignments, and discussions, students gained interface knowledge and understanding of heuristic evaluation principles. The first two weeks of readings consisted of background information about Second Life. During this time, students were instructed to take a Second Life tutorial and explore help menus. Additionally, students brought problems they did not understand about Second Life to a discussion board called “Second Life Q&A.” Through these methods, students learned about the philosophy of Second Life (i.e., avatars, Linden dollars, purchasing, building communities, selling, etc.), avatar customization, networking, avatar maneuverability, communicating, and using help screens. They were also given time (4 months) to become familiar with Second Life before being asked to evaluate it. This immersion in the interface is unusual in heuristic evaluation methodology, but supports the constructivist approach to learning. Though they could not gain expertise in heuristic evaluations in one semester, students learned the basics about the process through assigned reading materials and online discussions. This way, they could discuss usability problems before identifying them as such.

Standardized Time

All student evaluators were given an equal amount of time to learn Second Life and complete the checklist. They could begin on August 28th, 2007 but needed to have the checklist and report completed by December 8th, 2007.

Benchmark

The benchmark used to compare student evaluators is a collation of all the unique and valid heuristic issues identified by the instructor or students. Skills were assessed by counting the frequency of issues each student identified and by determining whether the student could explain why the heuristic violation could be harmful. For instance, under the heuristic, “User Control and Freedom,” one student commented on the fact that after going back to a previous scene, users must remember what happened in the old scene because there is no real undo. He stated, “There is no undo function as such, but mistakes can be recovered from if their previous selections are remembered. The problem with this is that people should not be required to hold too much in short-term memory.” This manifested a clear understanding of the HCI issue involved.

Skills Assessment

After students completed the checklist, they reported their findings in the usability reports. Valid problems identified by students or by the instructor were listed as usability issues. Though the reports were the primary sources used to assess student skills, the professor reviewed the diaries and checklists and interacted with Second Life to understand the students’ knowledge. The checklist provided organization for student reports and was the source document for the issues each student found with the Second Life interface. The diaries allowed for insights that informed the write-up.

The instructor identified five issues that an evaluator might easily detect as usability problems. It was believed that identification and understanding of these would establish a minimum amount of usability knowledge. These included lag time, unhelpful sounds, confusing messages, avatar maneuverability, and reversal of actions. Ten points were allotted for each of the above problems identified by students. Five points were given for other valid usability problems stated by students in their reports. The remaining issues identified were lack of white space, misnomer of world menu items, inappropriate metaphors, and distracting elements. If students could also explain which heuristic was violated and potential effects of the violation they were given 15 additional points.

Previous | Next