upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Meta-Analytical Review of Empirical Mobile Usability Studies

Constantinos K. Coursaris and Dan J. Kim

Journal of Usability Studies, Volume 6, Issue 3, May 2011, pp. 117 - 171

Article Contents


Recommendations and Conclusion

To the best of our knowledge, this research is the first analysis of the contextual factors and measurement dimensions investigated in the empirical body of knowledge of mobile usability studies published to-date by leveraging a proposed qualitative review framework for mobile usability. The results described earlier enhance our understanding of mobile usability research considerations and serve as the basis for a research agenda in this field. This domain would benefit by having a further emphasis placed on the complexity of contextual usability and answering such research questions as those within and/or between each of the following areas:

Technology: Beyond the interface—how do mobile technology components beyond the interface (e.g., network connectivity reliability, memory) impact the usability of mobile devices?

The results of the meta-analytical review of empirical research on mobile usability identified 31 usability-related measures. The main usability measures studied in mobile usability studies are efficiency, effectiveness, and satisfaction, which are actually consistent with the standard diminutions of other general usability studies (Brereton, 2005; Hornbaek & Law, 2007; Nielsen & Levy, 1994). However, these usability dimensions are more important in mobile applications and technologies because of the inherent characteristics of mobile devices, including small screens, low display resolutions, limited input methods, difficult-to-use interface, and many others.  Moreover, the three core dimensions of mobile usability measurements (i.e., effectiveness, efficiency, and satisfaction) reflect the ISO 9241 standard making a strong case for its use in related future studies. The use of this standard would allow for consistency with other studies in the measurement of general usability (Brereton, 2005; Hornbaek & Law, 2007; Nielsen & Levy, 1994).

Beyond the benefit of a standard view of usability, three key findings emerge from the above data. First, any single peripheral usability dimension was measured in fewer than 8% of the studies reviewed. Second, accessibility, in the context of vulnerable populations/disabled users, appears to be one of the most underserved research areas having been studied only twice in this set of 100 mobile usability studies reviewed. This observation may come as a surprise, given the growing popularity of accessibility research in less conventional (e.g., non-IS, non-peer-reviewed) publication outlets, and the increasing levels of legislative support and community interest. Further exploration of this construct, including its role with the remaining usability dimensions, is warranted. Third, aesthetic/hedonic constructs were studied in just 2% of empirical mobile usability studies, even though there is support for the effect of such factors on performance and satisfaction (Coursaris, Swierenga, & Watrall, 2008). These findings in turn call for a critical review of the current operationalization of usability as several dimensions are not captured in the international standard defined by ISO 9241 in 1998.

After more than a decade’s worth of research that centers on the standard usability measures articulated by ISO in 1998, our understanding of their inter-relationships is mature. The domain could arguably benefit by extending the defined core by considering a subset of the peripheral dimensions so as to allow for an even deeper understanding of mobile usability. Adding to the earlier research agenda, the following measurement considerations are outlined for future research: (a) accessibility—increasing research in this area may improve the usability of products and services for often overlooked audiences; (b) hedonics—which factors impact the aesthetic appeal of a mobile device or service, and how do they impact usability?; and (c) usability—what are the relationships between various usability measurement dimensions? Should usability be redefined to reflect additional utilitarian and/or hedonic dimensions?

This study offers several contributions and implications for both researchers and practitioners. On the academic level, first, this breakthrough meta-analytical research is the first attempt, to our knowledge, to offer a comprehensive view of usability dimensions found in empirical mobile usability studies. Second, the identification of a common measurement metric with a review framework would support a future quantitative analysis of mobile usability studies at the construct level (i.e., a meta-analysis of measured usability dimensions in a mobile setting). In turn, this could offer a unified view of empirical mobile usability studies. We hope that the framework and the findings of this study will be used as the basis for continuing research that aims to enhance our understanding of mobile usability considerations and measurement.

This study also provides a couple of important implications for practitioners. First, this study summarizes the existing mobile usability research findings and organizes them based on a set of usability contextual factors and measurement dimensions using a comprehensive mobile usability framework.  The results of this study encourage practitioners to pay more attention to the key contextual factors and mobile usability measurement dimensions when they develop their mobile products and/or services. Second, because the current mobile usability evaluation process is more of a “fuzzy art” without a structured framework and there is a need for a more structured approach to evaluate mobile usability, the mobile usability framework identified by this study can be used during a usability evaluation of mobile products and/or services.

As with all research, this study comes with the caveat of the following limitations. First, even though the authors searched intensively for all possible research articles of empirical mobile usability studies, the case may be that relevant articles were omitted in this process. Second, even though the meta-analysis of this study followed the procedures suggested by Glass et al., (1981), Lipsey and Wilson (2000), and Rosenthal (1991), some subjective decisions were made when two mobile usability dimensions were collapsed into a single measure. Although arguments were given, this could be a limitation of a subset of the reported results.

Beyond the benefit of a standard view of usability, an important opportunity for future research arises from the data in Table 1. Accessibility appears to be one of the most underserved research areas. Again, this observation may come as a surprise, given the growing popularity of accessibility research in less conventional (e.g., non-IS, non-peer-reviewed) publication outlets, and the increasing levels of legislative support and community interest. Further exploration of this construct, including its relationship with the remaining usability dimensions, is warranted.

In closing, it is hoped that the above findings and the suggested research agenda will stimulate further research in this domain, the results of which expand both the scholarly body of knowledge, but also have direct and tangible benefits for everyday users of mobile technology.

 

Previous | Next