upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Switching Between Tools in Complex Applications

Will Schroeder

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 173-188

Article Contents

Interpreting the timelines

With practice, we can pick out problem areas that might have been overlooked as the test was running. For example, the brief visit to the documentation that user 6 made at about 22:30 into the plotting task (see Figure 1) resulted in misleading information, requiring another visit to the documentation a minute later. Of course, the cause of this problem can't be read off the timeline. But investigating brief visits to documentation is almost always valuable from a usability viewpoint. When the trip is successful you can take the time to figure out what worked. We see in the timeline when they must repeat the visit (usually because they have not been able to bring back enough information).

Long visits to helper tools (row 1) may also be of interest. For example, why is user 6 spending so much time in the Array Editor? In addition, the statistical data on which this picture is based will quickly show whether other users also spend large amounts of time with a relatively non-productive tool. But the main value of the timelines here is their depiction of different users' tool strategies.

All of this paper's key questions appear in Figures 1 and 2. The tools (level 5 in the timeline) user 6 applied to the plotting task are designed for plotting. User 6 completed 10 of 13 segments of the task with them. User 2 applied general-purpose programming tools to the same task, completing 5 segments in about the same length of time.

We might say that user 6 was where we wanted him to be (in terms of tools), and user 2 was not. Yet user 2's Standard Usability Scale (SUS) score was almost twice that of user 6 (72 to 40). In relative terms, then, user 2 was comparatively unaware of how poorly the tools he chose were performing. To compound the problem, user 6 did much less well on the programming task than on the plotting task, finishing only one segment, yet his SUS score for that task was ~50% higher than for the plotting task (57.5 to 40).

The timeline view enables flagging of critical points in the testing (previous section), and grouping of workflows by patterns or styles, as shown here and in the discussion. The following sections present the limitations of the current version.

Drawbacks and advantages of the logged data

This automatically captured data provides us with information about users' workflow which we could obtain in no other way, but it is neither perfect nor complete. We can make most effective use of it by remaining alert to the following limitations:

While these limitations may lead to misinterpretation of individual usages and switches, they are more likely to be associated with statistical outliers-overlong dwells in the first case, and very rapid changes in the second.

These problems should be seen against the main advantage of the method, which is to gather and manage large amounts of data during conventional usability testing with little or no additional effort. Large amounts of data are essential to this investigation in order to identify and compare patterns of behavior that are, even without the noise generated by inactive periods and rapid focus shifts, statistical. This logging, counting, and plotting extend conventional usability practice by enabling consideration of new problem areas.

Previous | Next