upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Comparing Computer Versus Human Data Collection Methods for Public Usability Evaluations of a Tactile-Audio Display

Maria Karam, Carmen Branje, John-Patrick Udo, Frank Russo, and Deborah I. Fels

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 132 - 146

Article Contents


There are many informative results obtained through this study. We discuss these in order of their appearance in the public usability domain.

System Reinforcement

There are several issues to consider before taking a system out into a public domain. First, it must be at a stage of development where it can be reinforced and safely used by the general public. Some technology may be too fragile or unsafe for this type of environment, and given the extent of the users that can potentially access this technology, safety is a major concern. While there are many systems that have been developed in the research lab, it is essential to assess the feasibility of upgrading and re-enforcing these devices before spending the time and resources to prepare them for a public domain. Before taking a system into the public usability domain, we recommend that feasibility studies or other laboratory work be conducted to ensure the potential of the system for future developments.

Universality Means for All Users

While it is possible to exclude many people from a system that is designed for universal access, conducting a public study may reveal problems that specific user groups may have with the system. In our case, we found that older participants did not enjoy the actuators or the air jets, and while we thought that these were important elements in the translation of sound into tactile sensations, results from our study showed otherwise. One approach to modifying the system without having to completely redesign these features is to offer the option to turn off one or more of the sensations as the user requires.

Ethics Approval

Another interesting aspect of this study involved the ethics application we submitted before conducting this research. Typically, studies involving human participants require approval from the ethics committee of a university. However, in our case, we were only interested in people’s feedback to the system, rather than in specifically assessing a user’s ability to use software. As a result, the ethics review board did not require a formal ethics application. However, they asked us to follow good ethical practices; a brief description of the research was provided to all participants including a discussion of voluntary participation and the right to stop the process at any time. In the computer-based survey, a short description was provided at the front end of the survey. Participants could abandon the computer survey at any time and it would simply time out and return to the beginning page.

Additional constraints from our ethics board stated that we could not record the voice or image of any user, and that we must ask permission from parents or guardians before approaching any children under the age of 10.

Given the nature of our study, the ability to obtain user feedback from the public domain greatly increased the opportunity to present our work to a large number of users and to gather valuable feedback for longer periods of time, without having to first obtain written permission. This will be included as part of the methodology we are developing for conducting future usability studies in the public domain.

Previous | Next