Flexible Hardware Configurations for Studying Mobile Usability
Antti Oulasvirta and Tuomo Nyyssönen
Journal of Usability Studies, Volume 4, Issue 2, February 2009, pp. 93-105
Article Contents
Requirements
To demonstrate how varied the requirements for a mobile usability lab can be, and as a further argument for the swiss army knife approach, let us analyze the following three evaluation scenarios:
- Usability testing of a mobile group media application. Goal: Measure user performance in basic communicative tasks, comparing against a pre-defined baseline of errors and task completion times.
- A real-world longitudinal study of a mobile collaborative awareness application. Goal: Understand real-world use of the system in a team of information workers.
- A comparative study of mobile map representations. Goal: Assess which representation, 3D or 2D, is better for mobile maps in the task of locating buildings in a city centre.
For the first evaluation scenario, a setup that records only the user interface might be enough. The two latter scenarios require at least some track of what happens in the user's environment. The third scenario is the most demanding in this respect. Ideally, it requires systematic second-by-second analysis of both bodily (turning of body, deployment of gaze, use of hands; analyzed post-experimentally from video tapes) and cognitive (verbal protocols) strategies, in addition to analysis of events at the interface and in the environment. Experience with several analogous studies of mobile systems has helped us gain perspective to mobile usability labs and spell out the following general goals:
- Mobility. The system moves with the user, capturing interaction reliably wherever and whenever it takes place both in indoor and outdoor contexts of use.
- Captures embodied interaction. The system captures both bodily and virtual components of interaction as well as environmental events that may have an influence on those aspects of interaction that are under scrutiny.
- Unobtrusiveness. The system does not itself bring about direct or indirect changes that would cause bias or distortion in ecological validity.
- Multi-method support. The system does not limit the researcher to one source of data.
- Redundancy. The system has multiple data capturing mechanisms, so if one method or source fails then the data is not lost.
- Quality control. The system allows the experimenter to be aware of the reliability of data captured both during and after an experiment. It can help the experimenter answer questions such as what caused missing data, what situations were data gathered, how reliably the data corresponds to the actual situations it comes from, and so forth.
Let us explain the background for these requirements. First, we took as our starting point that the core of mobility is that it foregrounds the changing relationship between the user and the environment. It raises new constraints and resources for interaction from context, and it allows users to involve and utilize new contexts. For the recording system it means that it should be able to record any significant action or event in context. In practice, depending on the situation, this may include near bodyspace of the user, distant and proximate physical objects that a user interacts with, ad hoc environmental events, as well as deployment of gaze. The view that the core of mobile human-computer interaction is in the triad environment-user-computer demands flexibility in the placement of cameras as well as their division among the environment, the user, and the device.
Second, a threat to experimental validity is that the system itself affects the phenomenon under study, for instance due to its (a) physical qualities-e.g., the weight of the system causing fatigue, (b) ramifications to the user's processing of the interface-e.g., a camera occluding the mobile device, or (c) social consequences-e.g., a hat not being acceptable indoors. Camera types, positioning, and form are crucial qualities that affect these problems. Another factor is the moderator. The presence of a moderator may also affect events and is not always desirable. This is yet another argument for modularity and flexibility for the selection of cameras and their positioning. The cameras should enable researchers to conduct studies without a moderator, preferably without sacrificing the ability to record in an environment. One solution is the utilization of surveillance cameras.
Third, the environment is not only of interest as such, but it also introduces noise, unexpected events, and it is the cause of technical unreliabilities. For example, due to a user walking very quickly, it may happen that the moderator is unable to reliably capture events in the front bodyspace of the user. These problems call for (a) redundancy in recording and (b) online quality control. The former can be addressed if the moderator can place "just in case" cameras to augment and back up the primary ones. The latter can be supported by providing a real-time copy of the A/V stream for the moderator.
Flexibility in practice
To address these requirements, we aimed for flexibility in the following four qualities of the system:
- Camera types. At the moment we have one model, "a sugar cube" (17 x 17 x 17 mm), but in the future a smaller "minitube" type (45 x 7 x 7 mm) will be explored.
- Camera attachment devices. The cameras attach using a pin, a shallow cell phone shell (Figure 4), a necklace (Figure 2), and a pole (Figure 2). With a pin we will be able to attach a minitube to a hat as in the LiLiPUT system (Schusterisch et al., 2007).
- Wires. Both wireless and cable transfer are enabled. Furthermore, by providing cables of different types and lengths, we enabled the positioning of user-worn, non-camera components either to a backpack or to a belt.
- Offloading cameras to the environment. We implemented an option to utilize environmentally placed cameras.