upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Meta-Analytical Review of Empirical Mobile Usability Studies

Constantinos K. Coursaris and Dan J. Kim

Journal of Usability Studies, Volume 6, Issue 3, May 2011, pp. 117 - 171

Article Contents


Introduction

Mobile devices are becoming increasingly popular, having already reached over one billion mobile subscribers. A recent forecast by the UMTS forum (2005) estimated that the global number of subscribers will be between 1.7 to 2.6 billion for mobile voice and 600 to 800 million for mobile data.  As consumers’ technology fears and adoption costs are reduced, mobile devices are approaching “mainstream” status around the developed world. Mobile devices propose increasing value to consumers found in “anytime, anywhere, and customized” connectivity, communication, and data services.

Although progress has been made in terms of technological innovations, there are obvious limitations and challenges for mobile device interfaces due to the characteristics of mobile devices (i.e., the size of small screens, low resolutions of the displays, non-traditional input methods, and navigational difficulties; Nah, Siau, & Sheng, 2005). Therefore, usability is a more important issue for mobile technology than for other areas, because many mobile applications remain difficult to use, lack flexibility, and lack robustness.

Research Motivation and Objectives

Usability has been the focus of discussion (Venkatesh, Ramesh, & Massey, 2003) and described by varying definitions (Nielsen, 1993; Nielsen & Levy, 1994; Shackel, 1991) in both academia and industry for a long time. Many of these definitions proposed that the central theme of usability is that people can employ a particular technology artifact with relative ease in order to achieve a particular goal within a specified context of use.  The turn of this century marked an increased focus on mobile usability studies for research in the field of Human Computer Interaction (HCI). Although a considerable volume of research on general usability exists, due to the novelty of mobile technology relatively few studies have been conducted focusing on mobile usability. Even worse, only 41% of mobile usability papers are empirical1 in nature (Kjeldskov & Graham, 2003). Moreover, there is no qualitative study on the usability dimensions considered in such mobile studies. Thus, our research aims to fill this gap and in doing so will also provide a roadmap for future mobile usability studies that will be of value to this relatively young research area. Specifically, this study addresses the following research question: What are the key formation and evaluation dimensions of usability in mobile technology usability studies?

To this end, this paper describes the qualitative review of more than 100 published empirical mobile usability studies. First, following a brief review of a usability evaluation framework in a non-mobile context, a framework of contextual usability for mobile computing2 is presented. Next, by using the proposed framework a qualitative review of empirical mobile usability studies is presented along with a discussion on the taxonomy used during the coding in this study. The results emerging from the comprehensive review of mobile usability studies are then presented, which include (a) the contextual factors studied, (b) the core usability dimensions defined and measured, (c) the peripheral usability dimensions explored, and (d) key findings in the form of a research agenda. Finally, this paper discusses the contributions and limitations of the research.

Literature Review and a Mobile Usability Framework

Usability studies have their roots as early as the 1970s in the work of “software psychology.” Over time, the focus of this body of research has shifted and most recently centered on the relevance of context of use for usability. The concept of context of use,as it relates to usability, emerged out of the work of several scholars (Bevan & Macleod, 1994; Shami, Leshed, & Klein, 2005; Thomas & Macredie, 2002) who attempted to identify additional variables that may impact usability. Varied situational contexts will result in emerging usability factors, making traditional approaches to usability evaluation inappropriate. The significance of this area emerges from its importance in yielding a reasonable analysis during a usability study (Maguire, 2001; Thimbleby, Cairns, & Jones, 2001). Furthermore, during the evolution of HCI mentioned above, the conceptualization of usability has varied extensively. The broad set of definitions and measurement models of usability complicate the generalizability of past studies at the level of the latent usability variable. Therefore, a usability study gains value when it is based on a standard definition and operationalization of usability. In the following section, we review a set of key approaches in evaluating usability as communicated in previous work.

Approaches to Usability Evaluation

Different approaches to usability evaluation have been proposed in different contexts such as websites (Agarwal & Venkatesh, 2002), digital libraries (Jeng, 2005), audiovisual consumer electronic products (Han, Yun, Kwahk, & Hong, 2001; Kwahk & Han, 2002), and many others. In the context of website usability, Agarwal and Venkatesh  (2002) presented five categories (i.e., content, ease of use, promotion, made-for-the-medium, and emotion) and subcategories (i.e., relevance, media use, depth/breadth, structure, feedback, community, personalization, challenge, plot, etc.) of website usability evaluation components based on Microsoft Usability Guidelines (MUG ; see Keeker, 1997). They also discussed the development of an instrument that operationalizes the measurement of website usability. Recently, employing the MUG-based model, Venkatesh and Ramesh (2006) explored an examination of differences in factors important in designing websites for stationary devices (e.g., personal computers) versus websites for wireless mobile devices (e.g., cell phones and PDAs). In the context of digital libraries, Jeng (2005) proposed an evaluation model of usability for digital libraries on the basis of the usability definition of ISO 9241-11 (ISO, 2004). The model included four usability evaluation comports: effectiveness, efficiency, satisfaction, and learnability. The satisfaction of digital libraries was further evaluated by the areas of ease of use, organization of information, clear labeling, visual appearance, contents, and error corrections.

In the context of audiovisual consumer electronic products (e.g., VCR, DVD players, etc.), Han et al. (2001; Kwahk & Han, 2002) suggested a usability evaluation framework that was similar to the subsequent work of Hassanein and Head (2003). The framework consisted of two layers: formation of usability and usability evaluation. The formation of usability layer had four contextual-components (i.e., product, user, user activity, and environment) that were well accepted as the principal components in a human-computer interaction upon which good system design depends (Kwahk & Han, 2002; Shackel, 1991). The usability evaluation layer was organized with three groups of variables: design variables (i.e., product interface features), context variables (i.e., evaluation context), and dependent variables (i.e., measures of usability).

Interestingly, there is no usability evaluation framework that yet exists in the context of a mobile computing environment. We believe it is a critical omission and an important topic warranting investigation. The next section looks at the key formative factors of usability as explored in contextual mobile usability studies. From this review, we propose a contextual usability framework for a mobile computing environment.

A Contextual Usability Framework for a Mobile Computing Environment

The work of several scholars (Bevan & Macleod, 1994; Shami et al., 2005; Thomas & Macredie, 2002) who attempted to identify additional variables that may impact usability and subsequently adoption, led to the conceptual emergence of context of use (herein referred to as context)as it relates to usability, also referred to as contextual usability.  Several frameworks encapsulating context have been proposed (Han et al., 2001; Lee & Benbasat, 2003; Sarker & Wells, 2003; Tarasewich, 2003; Yuan & Zheng, 2005). While there may be other usability frameworks that attempt to capture the essence of context, the models cited here provide a representative set of work in this area. From these we adapted the framework proposed by Han et al. (2001) because it offers considerable detail for each dimension they identified.

On the basis of the discussion on approaches to usability evaluation and the framework proposed by Han et al. (2001) and Kwahk and Han (2002), we propose a contextual usability framework for a mobile computing environment. The framework is depicted in Figure 1 and contains three elements. First, the outer circle shows the four contextual factors (i.e., User, Technology, Task/Activity, and Environment) described earlier as impacting usability. Second, the inner circle shows the key usability dimensions (i.e., Effectiveness, Efficiency, Satisfaction, Learnability, Flexibility, Attitude, Operability, etc.). Third, the box on the top of contextual factors shows a list of consequences (i.e., improving systems integration, increasing adoption, retention, loyalty, and trust, etc.).

Compared to the framework proposed by Han et al. (2001) and Kwahk and Han (2002), there are several advantages of the suggested mobile usability framework. Although the previous frameworks proposed by Han et al. (2001) and Kwahk and Han (2002) are comprehensive, they are difficult to follow due to formation and evaluation dimensions being merged into one diagram. Thus, the suggested framework depicted in Figure 1 represents a simple yet direct way to identify and address the various contextual mobile usability dimensions. In addition, with its central focus on usability, it offers specific guidance on the implementation of any interface/interaction project along with potential outcomes.  

In addition, two modifications are introduced in terms of nomenclature for mobile contextual usability. First, “Technology” replaces “Product,” as this term helps conceive the system that a user may interact with a greater set of components, instead of simply the device or application itself. One example of this is found in the case of mobile usability where the inclusion of the wireless network is likely in addition to the mobile device (i.e., the product) when studying usability of a mobile product or service. Because mobile usability is mainly related to mobile technology, which continually improves the limitations of mobile interfaces and its applications, the technological factor of a mobile usability framework is an important and unique component that needs to be taken care of. Second, “Task/Activity” replaces “Activity,” as the former term appears more commonly in usability literature when describing the nature of users’ interaction with the technology. In addition, a list of consequences of usability was added to the framework as an output of usability evaluations. 

These four variables (i.e., user, task/activity, environment, technology) were used for the presentation of the qualitative review of previous empirical research3 that relates to the usability assessment of mobile applications and/or mobile devices. The benefit of using these variables for the literature review is found in both the structure it provides for the discussion to follow, as well as to help highlight any areas that are lacking investigation.

 


 

1Empirical studies deal with empirical evidence that is derived by means of observation, experiment, or experience. In this study, we further classified empirical evidence as survey, interview, observation, and device/server logs in either a lab, the field, or both settings, as well as focus groups.  

2Even though we mainly focus on mobile usability, our adapted framework can be used for usability studies in general. 

3Since this study focuses on mobile usability, we only reviewed empirical studies on mobile usability.

 

Previous | Next