upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Comparing Computer Versus Human Data Collection Methods for Public Usability Evaluations of a Tactile-Audio Display

Maria Karam, Carmen Branje, John-Patrick Udo, Frank Russo, and Deborah I. Fels

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 132 - 146

Article Contents


Designing and implementing an interactive system that offers universal access to music, sound, and entertainment to a diverse range of users poses many challenges. Once such a system is designed, however, it must be evaluated by as wide a range of user groups as possible in order to determine whether it actually achieves the goal of broad access. While laboratory studies have served as one of the most effective methods for assessing interactive systems with users, it can be difficult to access diverse populations of potential users. In addition, laboratory studies are often contrived and controlled, providing limited ecological or in situ validity, which may be highly desirable for a system that provides access to entertainment for the public.

The Emoti-Chair is a system that aims to provide universal access to the emotional expressiveness that music and sound can add to the entertainment value of a visual presentation for users (Branje, Karam, Russo, & Fels, 2009; Karam, Russo, & Fels, 2009). It uses a variety of tactile displays to create crossmodal representations of the emotional content expressed through the audio stimuli that accompanies visual media such as film or television. One important aspect of developing the Emoti-Chair is evaluating its usability and entertainment value in an appropriate venue where a variety of users can experience it.

Relying solely on laboratory experiments to evaluate the entertainment value or comfort of a system like the Emoti-Chair is problematic in that laboratory environments are not typically conducive to supporting relaxed, fun experiences for users. It is also difficult to attract a large and diverse population of potential users, including those with different hearing abilities, to a laboratory environment. Thus, real-world feedback from many potential users of the system may never be obtained during the development lifecycle. Also, it is often the case that systems that fare well in laboratory studies end up as commercial products without ever having been effectively evaluated in the actual interaction context they were designed to support. This is even more problematic for systems based on universal design principles (see Burgstahler, 2008 for a review), because there are so many potential user groups that must be accounted for in the design.

Testing interactive systems within a public space such as a public museum offers an alternative method for retrieving information from a wide range of users and their interpretation of the functionality and usability of a system within the context of the intended interactions, which can lead to improved usability studies in general (Christou, Lai-Chong Law, Green, & Hornbaek, 2009). While testing in a real-world setting can yield a richer set of user feedback, there are also many obstacles that must be addressed before new interactive systems can effectively be presented in a public space even for evaluation. These include ensuring that the system is robust, safe, reliable, and effective for use in an uncontrolled public environment. In addition, automatic data collection methods can be deployed to leverage the large numbers of potential users who visit public spaces to supplement the data that can be collected by human researchers who may not be available at all times.

In this paper, we present a report on a public usability study that was conducted at a public science museum with the Emoti-Chair. The version of the Emoti-Chair used in this study was extensively reworked to prepare it for display as the tactile component of an audio-visual (AV) presentation targeting global warming and the environment for high school children from the northern territories of Canada, and from a more urban setting within Toronto. We wanted to determine if the role of the chair in the exhibit was apparent to users, while evaluating the overall comfort, clarity, and context of the crossmodal displays for those users. We report on the efforts taken to prepare the chair for public display and on the two approaches (automatic and human facilitated methods of data collection) used to evaluate the effectiveness of the chair.

Usability for Universal Design

Universal design differs from conventional design practices in that designers are asked to anticipate the varieties of users and uses for their product that they may not have previously realized and/or discounted. Whereas conventional design practices encourage designers to address the needs of the anomalous average user, progressive designers who adhere to the principles of universal design realize that this mainstream user simply does not exist (Udo & Fels, 2009). As such, these designers understand that users have needs and preferences that differ, not only between subjects, but also over time (Rose & Meyer, 2002). Universal design theory proposes that a “user population” and “potential uses” of designs must be very broadly defined to include a variety of users and user abilities that would not be considered within the definition of average. Shneiderman (2000) argues that it is the responsibility of technology enthusiasts to “broaden participation and reach these forgotten users by providing useful and useable services” (p.87). In designing technology that is proactively built to ensure use by individuals with a wide variety of needs, designers give all users the ability to customize and individualize the way in which they use technology.

The Emoti-Chair was developed as an alternative avenue for translating audio stimuli, serving to simultaneously substitute or complement sound stimuli with movement and other tactile sensations. Musicians and artists worked with students to create an audiovisual entertainment experience for the Emoti-Chair that was indicative of the children’s understanding of rural versus urban spaces and the role of technology in society. Students who designed the original audiovisual presentations were given the opportunity to compose a tactile experience for the Emoti-Chair. The design process used to assemble the exhibit held true to many of the tenets of universal design, mainly in the inclusion of the original designers (the children) throughout all stages of the creation of accessible media. In addition, no special enhancements or interface alternatives (assistive technology) were required to enable people who were deaf or hard of hearing to use the system. The same access was provided to all users, regardless of their ability.

The chair was constructed to take into account as many of the principals of universal design as possible, although there were some shortcomings, which will be addressed in further research. For example, the design did not address the needs of wheelchair users who may not wish transfer from their own chair to the Emoti-Chair.

The Emoti-Chair

The Emoti-Chair was designed to present sound stimuli as tactile sensations to the body. Originally developed to assist deaf and hard of hearing (HoH) people in accessing sound information from films, the Emoti-Chair system is based on universal design principles, which aim to increase access to all forms of media for users of all abilities. The system is installed in a large comfortable gaming chair, which has been augmented with a variety of tactile displays aimed at communicating different elements of sound, including speech prosody, music, and background or environmental sounds as physical stimuli. These displays include motion actuators, air jets, and vibrotactile devices that are distributed throughout the chair. Each of these devices can be independently controlled by a computer system, offering the potential to develop a lexicon of tactile sensations that can represent audio events occurring in music, film audio, or live performances.

One of the theoretical contributions of the Emoti-Chair is the model human cochlea (MHC) (Karam, Russo, Branje, Price, & Fels, 2008). The MHC is a sensory substitution technique we use to present the musical component of sound as vibrations. The MHC distributes sound to arrays of vibrotactile channels that are embedded into the back, seat, and arms of the Emoti-Chair (see Figure 1). Our approach attempts to leverage the skin as an input channel for receiving musical signals and other sounds as physical vibrations. We placed the signals relating to music along the back and seat while signals representing human speech are presented along the arms of the chair. Background noises that are not related to music or speech are communicated using motion actuators and air gusts, which enable us to explore the sensory substitution of different types of sounds as unique tactile sensations. The Emoti-Chair uses the air jets and the physical actuators to represent other elements of sound such as wind blowing, earth shaking, or subways moving, although the mapping of sensations to sound is a topic for further investigation.

While this chair is primarily intended for use as an entertainment device, an additional function is to provide a platform for supporting research into the use of different tactile sensations as substitutions for sound. Through the different tactile elements included in the Emoti-Chair, it is possible to conduct investigations that can contribute to our understanding of the relationships between sound and touch. Each of the tactile devices is fully configurable and can be customized as required for the given application.

Furthermore, we are investigating the chair as an artistic contribution that can add a third modality, the sense of touch, to expressive art interactions that transform audio-visual (AV) displays into tactile-audio-visual (TAV) displays.

Figure 1

Figure 1. Emoti-Chair that was used in the museum exhibit.

Physical Preparation

To begin this study, several requirements had to be met to ensure that this novel interface would be appropriately integrated into the exhibit. First, because the chair would be exhibited in a public domain, it was necessary to reinforce the prototype to handle the volume of users who would be seated in the chair over the four months of the exhibit, which reached over 6,000 by the second month. This involved reinforcing and refining the prototype to ensure that it would be functionally and physically robust for the duration of the exhibit. The body of the chair was reinforced using a wooden structure. In addition, the hardware was upgraded and modified to account for the continuous use it would have to endure. The peripheral devices, including a large screen display, air supply, and Internet connection, were integrated into the infrastructure of the museum. All potential pinch points were eliminated and electronics access points were locked to minimize the safety risks.

Technical Preparation

One of the issues addressed for this public display was the need for remote administration and maintenance of the system. Because the Emoti-Chair was a research prototype, we had to account for potential support issues that could arise from extensive unsupervised use. Technical adjustments that were outside of the duties of the museum staff had to be handled remotely by our research staff. As such, our researchers ensured that they were able to remotely access the system and, in cases of emergency, log in and reconfigure the system in the event of a failure or crash. The entire system also underwent extensive testing and modification to ensure robustness, safety, and ease of administration in the event of any technical problems that occurred when our research staff were not on site. These problems included unexpected software and hardware crashes or issues that could potentially arise from such a high volume of users.

We note that an incidental benefit of conducting research in a public domain was the level of stress testing that the system received. To support the museum staff in administering the Emoti-Chair, we modified our software control interface to provide a large on-off switch that museum administrators could access to start or stop the display. Otherwise, the system was fully automated to run continuously and in synchronization with the AV component of the display. An infrared switch was placed on the chair to enable the system to be activated only when a visitor was seated. Because we were working with an early prototype, we anticipated that many problems would be revealed over the course of the exhibit. Our research staff could be contacted by the museum staff, who would call in the event of a system failure for remote administration of the system. Although the system was easy to use, the user interface and overall components were not fully developed for public use, and our research team spent many hours monitoring the system to ensure that the display was running as designed.

Content Production

The content creators were given an introduction to the functionality of the chair and its tactile sensations so that they would be able to create their part of the exhibit and incorporate the tactile sensations of the chair. This was an interesting process because most of the people on the content creation project did not have any prior experience with tactile displays or with the chair itself. We held a series of introduction and familiarization sessions with the contributors, which included high school students, musicians, teachers, and researchers. Everyone on the project had the chance to operate and experience all of the tactile elements of the chair, and each contributor was given a brief set of guidelines to use when integrating sound and video with the tactile display. This represented a creative component of the Emoti-Chair, which was a new experience for the AV producers collaborating on the project.

One of the challenges of this study was to determine how AV content providers would approach the inclusion of the tactile components in their presentations. Because we had a short period of time to introduce the content producers to the system, we used an ethnographical approach that enabled us to observe how the content producers would use the tactile display components and incorporate them into their work. The researchers on the project also benefited from this process as we had the opportunity to obtain firsthand experiences of how artists would approach the design of a TAV display. We considered several different approaches for designing the tactile element of the display, and we discuss some of these next.

Designing audio-visual-tactile content

Most of the people working on the project were very excited about the possibility of appealing to the tactile senses in addition to audio and visual; however, it was not clear how they would approach this as a design problem. To bootstrap the process, we suggested that they consider the energy usage as a metaphor for assigning tactile sensations to their AV content. Some of the students determined that the more energy use that was taking place in the video would be reflected by a stronger set of tactile sensations, while others were not sure how to map any of the tactile sensations onto the AV content of their exhibit. For the music displays provided in the arms and back of the chair, there was little confusion. Students simply directed the music component of their presentation to the tactile display. To assist students in creating the TAV presentation, we developed a preliminary set of guidelines to facilitate the design of tactile sensations by mapping them onto the visual events in the film, as described below:

Once the chair was prepared and the tactile content was choreographed, we designed the public usability study that would take place during the exhibit to gather initial responses from the public about this type of multimodal display. A picture of the chair as installed at the museum is shown in Figure 2. The large screen display was mounted from the ceiling and placed in front of the chair where it would be visible.

Figure 2

Figure 2. Children trying the Emoti-Chair at the Ontario Science Centre.

Previous | Next