upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Methodology for Measuring Usability Evaluation Skills Using the Constructivist
Theory and the Second Life Virtual World

Debra J. Slone

Journal of Usability Studies, Volume 4, Issue 4, May 2009, pp. 178-188

Article Contents


At a minimum, online interactive systems or software should ensure that users are kept informed of consequences of their actions, can select and sequence tasks, do not have to memorize a great deal, feel privacy is maintained, are not encumbered with irrelevant elements, can easily find information, have aesthetically pleasing and highly functional experiences that enhance the quality of their work, can customize the system to their needs, and feel that the system conforms to their expectations. Software is usable if it adheres to these established usability principles. Heuristic evaluation is an effective way to determine adherence.

Because expert heuristic evaluators help software developers avoid costly problems, mastery of usability skills is not only highly desired for software development, but essential. Relying on constructivist theory, this study presents (a) a means to teach and measure usability evaluation skills of graduate students using structured and unstructured assessment tools and (b) the use of diaries to facilitate reflection and deep learning of usability principles. The Second Life virtual world was the platform for this study. The study questions were as follows: Can student usability evaluation skill levels be measured using structured and unstructured assessment tools? Do diaries facilitate deep learning of usability evaluation skills?


The following sections discuss heuristic evaluation, the constructivist theory, and reflection and learning diaries.

Heuristic Evaluation

Heuristic evaluation, one of the most commonly used methods for determining the usability of a software system, detects a high number of problems using a minimal number of people. Employing established usability principles (or heuristics), evaluators study an interface and identify problems. The literature supports heuristic evaluations primarily because of the affordability and convenience (Desurvire et al., 1992; Doubleday et al., 1997; Gerhardt-Powals, 1996; Liljegren & Osvalder, 2004; Nielsen & Mack, 1994). Yet, heuristic evaluations have limitations. Compared to user testing, for instance, heuristic evaluations fall short of discovering many severe usability problems (Jeffries, Miller, Wharton & Uyeda, 1991). When inexperienced evaluators are used, there is a greater potential for applying the wrong heuristics, and in turn, the wrong solution to problems (Nielsen & Landauer, 1993). Also, heuristic evaluations produce a high number of false positives (Bailey, 2002; De Angeli, et al. 2003; Kantner & Rosenbaum, 1997; Molich, 1998). Typically, experts conduct heuristic evaluations. However, studies have found successful heuristic evaluations conducted by system users after receiving training (Lowry & Roberts, 2003; Nielsen, 1993).

Constructivist Theory

Studies of online learning report a paradigm shift from information transmission to active participation models. One such model is the constructivist approach whereby students construct their own understanding of concepts by becoming active participants, rather than passive listeners, in the learning process (Biggs, 2003; Bruner, 1973; Papert, 1990).

Though constructivists differ in how they define knowledge and learning (Gredler, 2001), most support the notion that knowledge is user constructed, emphasizing individual learning whereby students integrate new experiences into their knowledge base over time. "[Learning] requires self-regulation and the building of conceptual structures through reflection and abstraction" (Von Glasersfeld, 1995, p.14).

Mayes and Fowler (1999) conceptualized the learning process as follows: (a) learning evolves from individualized understanding through performance of actual tasks, (b) frequent feedback facilitates learning, (c) learning progresses in a series of stages, and (d) learning takes place through both personal and social constructs. This learning process is described in three levels: conceptualization, whereby students are exposed to information, ideas, and concepts; construction, in which students apply what they know to a task; and dialogue, where students test their conceptualizations through personal reflection and engagement with others.

Reflection and Learning Diaries

Constructivist learners use experience, observation, and reflection to understand concepts and generate new ideas (Kolb, 1984). Of these, personal reflection, an active process of self-awareness and review of one’s experiences, allows for the acquisition of personal knowledge and development of personal views on the learning process or a specific issue (Gibbs, 1991; Kolb, 1984; Stewart, Keegan & Stevens, 2008). Diaries, or learning journals, facilitate reflection in a self-directed environment (Crème, 2008; Honey & Mumford, 1989; Mansourian, 2008; Pavlovich, 2007). Diaries allow for regular and deep reflection of experiences, aid active processing of knowledge, help students determine their progress, and assist in identifying gaps in knowledge. Barclay (1996) successfully used learning logs in the form of diaries. Pink, Cadbury, and Stanton (2008) used e-portfolios with similar results.

Using qualitative and quantitative measures, Gilar, Ruiz, and Costa (2007) determined that more knowledge was acquired using diary-based assessment than an inventory-based learning strategy assessment. Using the structured diaries, participants recorded temporal elements as well as every day activities, activities related to the task of learning, and activities aimed specifically at the learning of the task. The inventory-based assessment evaluated learning strategies using a questionnaire.

In an online environment, reflection and action are crucial to learning, allowing students to become proactive in the learning process. Hence, critical thinking, introspection, interaction, and synthesis of knowledge are sought. This requires resources beyond lectures.

Previous | Next