upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Comparing Computer Versus Human Data Collection Methods for Public Usability Evaluations of a Tactile-Audio Display

Maria Karam, Carmen Branje, John-Patrick Udo, Frank Russo, and Deborah I. Fels

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 132 - 146

Article Contents


The Study

The study we conducted focused on several aspects of the chair and its associated tactile sensations. These included learning whether visitors to the exhibit noticed and comprehended the relationship between the exhibit and the tactile sensations of the chair and exploring which of the tactile devices were most appealing to different users. We also wanted to explore the automation of data collection as an alternative to having only human researchers administering the questionnaires. We were interested in determining whether there were any significant differences in the two data collection approaches to obtaining user feedback on the system, towards further developing a methodology for conducting public usability evaluations.

Additional criteria for designing the questionnaires included limiting the amount of time and effort that was required from the museum visitors, so we only focused on the most critical aspects of the exhibit and the TAV for this study, and rewording the questions to be accessible to a young child’s reading ability.

Questionnaire Design

The key features of the display we hoped to examine through the questionnaire included understanding whether users could perceive the relationships that linked the tactile, visual, and audio elements of the display with the exhibit, assessing the level of enjoyment users derived from the experience, and the level of comfort the chair would provide. We modified an existing questionnaire used for in-laboratory evaluations (Karam et al., 2008) to fit in with the exhibit and the issues being explored in the public usability environment.

As this was our first attempt at conducting public usability studies, we conducted several iterations of pilot studies that led to the final version. This study was primarily aimed at hearing individuals who could provide us with a baseline understanding of the connections that could be drawn between the sound and the tactile sensations. There were a total of 25 questions in each questionnaire.

The questionnaires asked users to indicate their sex, age group (under 10; youth 11-19; adult 20-64; senior, over 65), which presentation they watched, to rate their comfort and enjoyment of the chair, the relationship between the chair and the exhibit, and the chair’s fit with the exhibit. We also asked about the amount of background information participants read before entering the exhibit, as well as preference and enjoyment ratings for each of the tactile devices.

Two methods were used to collect questionnaire data from museum patrons: researcher administered and computer administered.

Researcher-Administered Questionnaire

The researcher-administered questionnaire presented the same questions as the electronic version. Researchers wore badges to indicate their name and university affiliation. After a participant experienced the Emoti-Chair, a researcher would approach them to see if they would be willing to answer a few questions to provide feedback on the exhibit and help us improve the design of the system.

We interviewed participants of all ages, but asked permission before approaching children. Most of the participants were eager to take part in the study; this may have been due to the type of museum (science oriented), where people were pleased to contribute their feedback towards improving the technology. Researchers were present at the museum three days a week for a period of two months and obtained a large number of responses; however, many of the responses were gained during the pilot study and were not included in our analysis.

Self-Administered Questionnaire

The self-administered questionnaire was presented using a PC computer and 15-inch touch screen monitor. The screen was mounted on the wall located behind the Emoti-Chair installation. The self-administered questionnaire was identical to the research-administered questionnaire, but the format was modified for the touch screen. The questionnaire application was developed in C# using Microsoft Visual Studio 2008 and a mySQL relational database. Each question was presented on a single screen, with the question displayed in large text at the top of the page and a large button below to represent possible responses, as shown in Figure 3. Additions were also made to the digital version of the questionnaire to make it more suitable to the touch screen format and to account for potential problems that could arise in this public domain. For example, the visual layout was designed to ensure that participants could easily distinguish the question text from the answer buttons.

Also, we implemented a monitoring system that could help us determine when there were problems with a particular session. For example, to determine if a user abandoned their questionnaire session, we used a timer to monitor the system at the start of each question. If the timer reached 30 seconds without a response, the questionnaire was reset. The session was terminated and marked as incomplete. If the participant responded within the 30-second timeframe by selecting one of the button responses, the timer was reset to 0 and a new question was displayed on the screen. After each response button was pressed, its value was saved in an SQL database on our remote server, and the screen was updated with the next question. The survey ended when all questions were answered. The computer running the survey was connected to the Internet so that data could be uploaded to our remote server on a regular basis. Maintenance could also be performed remotely when the survey application or computer failed. We also added an introduction screen to explain what the questionnaire was about and to check if participants had already tried the chair. If they did not, they were asked to return and complete the questionnaire only after trying the chair. This was not a problem in the human-administered version.

Participants

The study was focused on evaluating the system with hearing individuals. The participants who took part in the study included children under the age of 10, seniors over age 65, as well as people between these ages. All were visitors to the science museum, and all took part in the study based on their interest in the display.

2

Figure 3. Screenshot of one question from the self-administered questionnaire

Previous | Next