upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Comparison of the Usability of a Laptop, Communicator, and Handheld Computer

Piia Suomalainen, Leena Korpinen, and Rauno Pääkkönen

Journal of Usability Studies, Volume 5, Issue 3, May 2010, pp. 111 - 123

Article Contents


Methods

The following sections discuss the subjects, the test environment and tasks, the questionnaires, the observation of the subjects’ work posture, the protocol, and the data analysis used in this study.

Subjects

The subjects were students and personnel of universities recruited by an open advertisement. All willing persons were included in the study. There were 25 subjects, 14 (56%) females and 11 (44%) males. The oldest was 48 and the youngest was 20. Over half of the subjects were between 20 and 24, but there were also three over 40. The medium age of the subjects was 26.7. The age and sex distributions for the 20 to 24 age group were nine females and five males, for the 25 to 29 age group there were two females and four males, for the 30 to 34 age group there were one female and one male, there were no subjects in the 35 to 39 age group, and there were two females and one male in the 40 or over age group. Eleven of the subjects wore eyeglasses. After the researcher presented the study, they signed voluntary written consent (agreement) forms before the tests started.

Test environments and tasks

The test devices were an HP XE3 laptop, a Nokia communicator 9110, and a Palm Tungsten T handheld computer. There was similar illumination in the room for each device. All devices were on a similar table and in a similar position. Figures 1, 2, and 3 show the test environments for each device (laptop, Figure 1; communicator, Figure 2; and handheld computer, Figure 3).

Figure 1

Figure 1. The test environment for the laptop

kuvia2

Figure 2.The test environment for the communicator

kuvia1

Figure 3. The test environment for the handheld computer

The usability of each device was evaluated based on a writing task and calculating tasks completed by the subjects. The calculating and typing tasks were secured to the table beside each device. The writing task for the handheld computer used Graffiti letters because it was easier to write with them. The models of these letters were secured to the table above the handheld computer. The subjects were given a different typing task for each device. The tasks were taken from the book by Kettula (1988). In general, the same Finnish text was used when the subjects’ speed of writing was analyzed. After the typing task, the subjects completed eight calculations.

Questionnaires

The research material consisted of five questionnaires. The first questionnaire (Q1) included questions on background information. After using each device, the subjects were asked some questions about that specific device. Three questionnaires included questions on the usability of the devices (laptop [Q2A], communicator [Q2B], and handheld computer [Q2C]). After the subjects had completed all of the tasks, they answered a questionnaire (Q3) where they were asked to compare the devices. Questionnaire Q3 contained the same scale points, but also contained a section where subjects could provide open feedback and observations. Experienced instructors designed the questions for the questionnaires based on principles published in ergonomic instruction literature.

Observation of the subjects’ work posture

At the same time the subjects were testing the devices, an observer (researcher) was analyzing the subjects’ work posture using the table of physical loading by Andersen and Bjurvald (1994). The same person observed and recorded her findings for all of the subjects. There was only one test subject in a room, and the subject did each phase of work (see Figures 1 through 3) sequentially. (Figure 4 presents a timeline for the study.)

Protocol

After the subjects entered the room, they were introduced to the study protocol. They had 10 minutes to acquaint themselves with the devices. Then they were asked some background information (Q1). Figure 4 shows the timeline of the study.

Figure 4

Figure 4. The timeline of the study: laptop = test 1, communicator = test 2, and handheld computer = test 3; Q1 = questionnaire 1, Q2A= questionnaire 2A, Q2B= questionnaire 2B, Q2C= questionnaire 2C, and Q3= questionnaire 3

The subjects started the test with the laptop, then they tested the communicator, and finally they tested the handheld computer. The test order for the devices was based on how well known a device was. The tests started with the most well known device (laptop) and finished with the least known device (handheld computer). For each test, subjects were given five minutes to complete a typing task. The typing tasks were different for each device test. After each typing task, they had five minutes to complete eight calculations on each device. After each device test, subjects answered questionnaires (Q2A, Q2B, and Q2C). After they had completed all tasks, they answered the Q3 questionnaire.

Data analysis

Data from the questionnaires and the observations from the observer (researcher) were entered into a computer. The questionnaires also contained a section for providing open answers—feedback and observations. These were carefully read through and examined to see whether there were some valid points, any valid points were typed into the computer. Microsoft Excel was used to study the data.

The subjects evaluated the devices on the questionnaires using the following scale:

After the evaluations, we discovered a possible bias that some subjects may have placed a higher importance on the higher number (5), so we changed our scale as follows. (There is a more complete description of this change in the “Evaluation of methods” section in this paper.)

For the typing portion of each test, we registered the length of the written text and any typing errors made by subjects (e.g., wrong letter, missing space, and small letter when it was supposed to be a capital letter). All typing errors for a single device were summed together and then compared with the errors of the other devices. For the calculation portion of each test, we registered the calculation errors made by subjects. All calculation errors for a single device were summed together and then compared with the errors of the other devices. The statistical analyses for the typing tests and calculation tests were done using PASW Statistics 18 (formerly known SPSS Statistics). The differences were tested using nonparametric tests, Friedman’s 2-way ANOVA by ranks (3 samples).

Previous | Next