upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Switching Between Tools in Complex Applications

Will Schroeder

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 173-188

Article Contents


Results

At this time, we have refined generation of the timeline and automated some charts that summarize and compare data. However, the data so far is only a baseline sample. The acid test of the technique and approach will come when we run the same tasks under the same conditions with improved designs. We expect that differences in tool usage will be easy to study and evaluate using this approach. Meanwhile, the results to date do support some observations of interest and potential value. (This discussion is limited to the programming task, which Users 3 and 5 did not attempt.)

We asked each user to fill out the Standard Usability Scale (SUS) after the task, and we recorded the number of task steps each user finished. The users also supplied information about themselves, rating themselves expert (E) or novice (N), on level of skill and comfort with MATLAB (1-5 low to high), and whether they thought of themselves as programmers or not. They also reported the number of files they wrote per month, and years they had used the product.

Table 1. User Data from the Programming Task
User SUS Score Completed Self-Rating Experience Editor-Command Window Switches
Expert? Skill with MATLAB Comfort with MATLAB Program-
mer
Files/
Month
Years Using MATLAB
1 60 2 E 3 4 Y 1 6 61 (High)
2 82.5 1 N 2 4 N 7 0.5 17 (Middle)
4 87.5 5 E 4 5 Y 24 5 90 (High)
6 57.5 1 N 2 3 N 9 2 4 (Low)
7 45 0 N 2 3   1 8 6 (Low)
8 40 1 N 1 2 Y 2 0.5 9 (Low)
9 70 1 N 1 3 N 4 5 8 (Low)
10 67.5 3 E 3 3 Y 12 3 29 (Middle)
11 90 3 E 1 3 Y 0 4 0 (Low)
12 100 1 N 3 4 Y 15 1 28 (Middle)
13 70 2 N 3 5 N 5 1.5 18 (Middle)
14 80 2 E 1 4 N 20 0.2 26 (Middle)

No correlation between user scores and their self-descriptions was apparent, nor was it expected. However, it was possible to group users meaningfully by their tool switching behavior, which is the content of the last column. How this was done appears in Figure 3.

Tool switching styles of 11 users who completed the programming task are presented as point data connected by lines in Figure 3. This view accentuates user differences and similarities around individual transitions. We can immediately pick out users 6 and 13 moving between the Editor and the Documentation, user 1's frequent use of the Array Editor, and so forth. We can also separate the users into three groups based on their frequent switching in and out of the Editor and the Command Window (the first two data on the x-axis). We can see what this grouping implies by comparing the "styles" depicted by the users' timeline diagrams. (This grouping is analyzed in Table 2.)

Transition statistics by user for the programming task, the frequency of switching from Editor to Command Window suggested the division of users into three groups: High > 50, Middle 18-30, and Low < 10. (Not all users tried both tasks; user 2's log was corrupted.)

Figure 3. Transition statistics by user for the programming task, the frequency of switching from Editor to Command Window suggested the division of users into three groups: High > 50, Middle 18-30, and Low < 10. (Not all users tried both tasks; user 2's log was corrupted.)

Figure 4 shows a timeline from each of the three groups. There are other visible differences, but sample sizes are too small for anything but speculation at this point.

User 1 (High group, top), user 8 (Low group, middle), and user 13 (Middle group, bottom) working on the programming task.

Figure 4. User 1 (High group, top), user 8 (Low group, middle), and user 13 (Middle group, bottom) working on the programming task.

Previous | Next