upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Switching Between Tools in Complex Applications

Will Schroeder

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 173-188

Article Contents


Methods of observation

Studies of multi-tasking and task switching in the literature provide a framework for categorizing data and controlling the test environment. A general taxonomy for modeling multitasking (Wild et al., 2004) enumerates possible sources of error and distortion in our test environment and protocol. The test environment provides that test participants do not inhabit different roles (developer, manager, leader, or contributor), that there are no interruptions from outside the task, that there are no proximate external forces such as job or peer pressure, that there are no events hindering one tool in favor of another, and that there is no change in work environment. The following are the most important points to consider:

Compared to most multi-tasking studies, usability testing is clean-room control.

Learning about tool switching begins with an accurate picture of what actually happens. So, to perform the necessary basic studies, we need a technique that extracts and summarizes pertinent (to tool changes) data and metrics from testing and focuses on critical incidents with little or no "dog work" to encumber the practitioner's day job.

Logging and reporting

Microsoft used silent logging extensively and famously to gather design requirements for Vista (Harris, 2008). A web site, Hackystat, provides an open source framework that enables anyone to log their own activity on a remote server and retrieve regular reports.

Recording operating system focus and user input captures time spent with each tool and an indication of the activity. The log record's time signal indexes into a recording of screen activity and users' commentary (if any). Additionally, success in tasks and subtask steps can be measured using completion, time to completion, user satisfaction, and the System Usability Scale (Brooke, 1996; Tullis & Stetson, 2004). We use Techsmith's Morae®, which captures some tool switching information, to record screen and voice.

A Java applet logs window activity and user input from within MATLAB® capturing all activity that can be precisely distinguished. A Visual Basic for Applications (VBA) script on the back end of the logger produces timeline pictures of tool use and changes, and statistical descriptions (counts and charts) that summarize the patterns with exhibits such as those presented in this paper.

Previous | Next