upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Methodology for Measuring Usability Evaluation Skills Using the Constructivist
Theory and the Second Life Virtual World

Debra J. Slone

Journal of Usability Studies, Volume 4, Issue 4, May 2009, pp. 178-188

Article Contents


Results and Discussion

Comments about Second Life were overwhelmingly positive. In fact, reports, diaries, and checklists consisted of three times as many complimentary or descriptive comments than problems. In short, students could easily identify what worked. Yet, they also identified 9 usability issues, including ones also identified by the instructor. Beyond this, 46 issues were presented and discussed in the Blackboard forum but only 16 appeared in the final reports and diaries. Seven issues were determined to be false positives. A false positive is defined here as an issue that was identified by students as usability problems but were not. Table 2 lists the 9 valid usability related problems identified by students.

Table 2. Problems and Potential Problems Identified by Students
  Identified issues

*Lag time

There was a general agreement that lag time would not allow complex tasks to be completed within 12 seconds. Even for simple movements, lag time made response times inappropriate to the tasks.

*Avatar maneuverability

According to one student avatar movement is “like working a joystick. It takes practice.” New users ended up in trees and were given no prompts for getting out. One avatar became locked into a backward movement. User skills did not increase considerably between the first and second visits.

*Unhelpful sounds

About the sounds, one student stated, “There are sounds for almost everything…Sounds, like constant beeping, are not always helpful.” Sound is often used to differentiate between error messages and background conversation or gestures. A user performing an action correctly may mistake the beeping for something she is doing wrong.

*Confusing messages

Though selections within the drop-down menu are brief and familiar, some were confusing or inappropriate, such as the “Wear on” option. “Where,” one student asked, “does one ‘wear’ a unicorn?”

*Reversal of actions

When users go back to a previous scene, they cannot change their earlier choices; they are irreversible. According to one student, “There is no undo function as such, but mistakes can be recovered from if their previous selections are remembered.”

Lack of white space

Meaningful groups of items are not separated by white space. The text is often small and spaced tightly, making it difficult to read.

Misnomer of world menu items

Choices are not always logical. The Edit menu, for example, displays options to go to a friend’s and group’s lists, allowing for direct communication with them. While a user can edit these lists, this is not the main function of these selections. The World menu features selections for chat, navigation, and account information. None of these seem to generate “world” in the mind.

Inappropriate metaphors

Sometimes metaphors are not clearly understood for objects that indicate a script (i.e., floating dance machines).

Distracting elements

Nonessential elements sometimes distract. Users can control menus by minimizing windows with nonessential items. Novices, however, may find themselves searching for essential menus. At the same time, dialog boxes with background noise may prompt them for a selection in reference to some other task.

Instructor-identified usability issues

Table 3 shows results based on the valid issues students identified and those they discussed or explained in their final reports. This table shows that most students could identify all the problems named by the instructor and additional problems as well. Further, for several issues, a number of students could explain the heuristic violated and why it is important.

Table 3. Breakdown of Issues Identified and Explained
Identified issues # of students
identifying issue
# of students
explaining

*Lag time

9 of 9

8

*Avatar maneuverability

9 of 9

9

*Unhelpful sounds

9 of 9

8

*Confusing or unhelpful messages

7 of 9

6

*Reversal of actions

7 of 9

4

Lack of white space

5 of 9

4

Misnomer of world menu items

2 of 9

1

Metaphors as cues

3 of 9

3

Distracting elements

6 of 9

2

*Instructor-identified usability issues

Table 4 shows the point breakdown of skills. Students garnered more than two thirds of points possible for the identification tasks, suggesting that identification of the problems was a manageable task for students to master. Explaining the problem, however, was a bit more difficult for students. More than half of the points available were distributed for this task.

Table 4. Skills Assessment Point Breakdown

Points for identification

Points for explanation

Possible

Actual

Possible

Actual

630

490

1215

675

There were, however, some heuristics a majority of students did not identify in their reports and even fewer who could explain. The diaries provide clues to why students may not have explained the heuristic. In her diary, one student referenced the problem with the misnomer of the world menu. She wrote, “That [world menu] doesn’t make any sense, the menu, but I guess I’m the only one that cares about that. Picky me.” This was not repeated in the student’s report. The results in Table 5 show that the total number of usability problems listed for the diaries are more than for the reports. These findings suggest that students thought the issues bothersome but either could not explain how they might affect usability or thought they were not worth mentioning in their report.

Table 5. Identified Issues Listed in Reports and Diaries
Identified issues # of students
identifying
issue
# of students
explaining
Issues
identified in
diaries

*Lag time

9

8

9

*Avatar maneuverability

9

9

9

*Unhelpful sounds

9

8

8

*Confusing or unhelpful messages

7

6

3

*Reversal of actions

7

4

5

Lack of white space

5

4

3

Misnomer of world menu items

2

1

3

Metaphors as cues

3

3

6

Distracting elements

6

2

8

*Instructor-identified usability issues

Conversely, students identified 7 problems that were determined to be false positives. Kantner and Rosenbaum (1997) found that 43 percent of issues in their study were false positives. In the current study, the percentage was 44 percent. The false positives discovered by inexperienced students in the current study indicate that there is a greater possibility of applying the wrong heuristics to a particular problem. Table 6 identifies the issues determined to be false positives.

Table 6. List of False Positives
List of false positives # of students
identifying issue

“[It’s] like a game, so no one can really get any work done.”

1

“Walking backwards does not always reverse walking forward.”

1

 The system performs “…actions too quick to cancel.”

1

“There are too many unavailable items in the tool bar because a membership is required to use them.”

1

Expressed sentiments of fear (or the “willies”) or danger from “predators” on Second Life.

4

Character customization of avatars is “ugly.”

1

“There is a condescending attitude of many of them [avatars] that make you feel uncomfortable, and the identity of the avatar is uncertain.”

 

1

Percentage of false positives of all issues identified

44%

Usability issues identified

9

False positives

7

Total issues identified

16

De Angeli, et al. (2003) determined several reasons for false positives, including observations based on the personal preference of the evaluators but that were not usability problems, problems that reflected the misjudgments of the evaluator, system defects due to hardware configuration, and unclear or confusing statements. As shown in Table 6, most false positives were attributed to the personal feelings or preferences of individual students.

Revisiting the research questions, this study suggests that heuristic evaluation skills can be measured using diaries along with a systematic process that includes established usability guidelines, standard instruments to assure soundness, and benchmarks to measure progress. Most students had a minimum amount of measurable usability evaluation skills in that they could identify heuristic problems. A fewer number had a deeper amount of knowledge in that they could explain the problems and the reasons why they were harmful.

The results of this study suggest that diaries provided a means for students to express their opinions and feelings about the usability process, Second Life, and heuristics. The detailed descriptions in the diaries beyond the usability checklist and report suggest that the diaries served as instruments for unstructured and deep reflection.

Previous | Next