upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Flexible Hardware Configurations for Studying Mobile Usability

Antti Oulasvirta and Tuomo Nyyssönen

Journal of Usability Studies, Volume 4, Issue 2, February 2009, pp. 93-105

Article Contents

First Experiences

To acquire feedback on the system, we deployed a recent study using the system for mobile maps. In this field experiment, we asked 16 participants to carry out localization tasks in a city center, with either a 2D or 3D mobile map (Oulasvirta, Estlander, & Nurminen, in press). No environmental cameras were used in this ultra-mobile study where 16 different city sites were visited. A moderator-controlled, high definition camera with a wide-angle lens was used instead.

Figure 6 depicts output data after integration of the video output (on the left) with a reconstruction of the interface we automatically built from the logs. Using custom-made software, the log files from the mobile map software were integrated and then manually synchronised with the four-channel video.

Figure 6. Sample data output from the first field experiment (Oulasvirta et al., in press)

Figure 6. Sample data output from the first field experiment (Oulasvirta et al., in press)

The results of the experiment showed that 2D was better in both localization and navigation tasks, in all dependent measures we deployed. The mobile lab enabled us to gather unusually rich data from verbal protocols to subjective workload measures and from video observations to integrated interaction logs. These data afforded for going beyond typically measured performance measures, like task completion times and errors. Particularly, we were puzzled by the finding that 2D was associated with not only faster performance overall but lower cognitive load than 3D. The 3D maps were typically thought to lower cognitive demands by making it easier to match what was seen in the surroundings to what was seen on the map (ego-centric alignment). Finally, we came up with the hypothesis that 2D users can better use their body to find alignment between the map and the visual scene. We carried out a second-per-second analysis of the video tape that proved this hypothesis right. We found that 2D users often tilted the device in their hands, used their upper body more, and were generally more efficient in their deployment of gaze (i.e., they found effective cues quicker). Video tapes in combination with interaction logs and verbal protocols thus enabled us to find an explanation that we could have not found otherwise. These results were analyzed to draw implications to a new version of the 3D map that is functionally and representationally closer to the 2D map while retaining some advantages of the 3D map. The full results are presented in Oulasvirta et al. (in press).

We were generally happy with the achieved quality. The final integrated video files were 2000 kbps MPEG-1 streams, with a resolution of 520 x 320, a frame rate of 25 fps and a MPEG-1 Layer 3 64 kbps mono sound track. The sound track was passed through a 3000 Hz low-pass filter to dispose of a high-frequency, whining noise. For this data, we were able to carry out manual coding of events with good inter-coder reliability (Kappa .75), even for quite subtle behavioral measures (such as turning of head, turning of device in hand, body posture, and walking), that were coded with one second accuracy. The freeware video annotation software InqScribe was used for coding.

There were a couple technical difficulties. We had to change recorder and battery type, and consequently re-ran three more subjects. The more persistent problems were due to environmental conditions and accidental events that hampered the use of one or more of the minicameras. Direct sunlight to the face camera, shutter adapting excessively to large contrasts in camera image, necklace camera temporarily occluded by clothes, random compression artifacts, and rain effectively prevented coding of some of the variables-particularly when these effected several cameras at the same time. These problems can be addressed in the future by camera selection, camera placement and attachment, and changing the video recorder.

We also noticed effects due to form and design. While our device-mounted parts include only two small minicameras weighing a few grams, they nevertheless affected how the mobile device was held in the user's hand, which created a bias to keep it more upright than normally. Second, while our setup attempted to minimize the visibility of cameras, particularly the pole on the device caught the attention of passers-by, although so sporadically that we did not assess this to pose a bias to the validity of the conclusions.

Previous | Next