upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Creating Effective Decision Aids for Complex Tasks

Caroline Clarke Hayes and Farnaz Akhavi

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 152-172

Article Contents


Figure 7 shows the time required by both intermediates and experts to rank the alternatives, and identify a "best" alternative.

Figure 7. Both intermediate-level and expert designers used more time when ranking alternatives using the computer decision aids.

Figure 7. Both intermediate-level and expert designers used more time when ranking alternatives using the computer decision aids.

It was evident from observations of the subjects during the experiment that the decision aid required more time than ranking alternatives by hand largely because of the data entry associated with using the decision aids. When ranking alternatives by hand, subjects simply sorted or numbered a stack of drawings that required relatively little time. However, they had to enter numbers for each alternative when using the decision aid. The fuzzy decision aid required more data entry (two numbers for each value) than the deterministic aid (one number for each value) which explains why the fuzzy aid required more time. However, spending more time with each alternative may also have encouraged subjects to think more carefully about their relative merits.

It may appear counter-intuitive that the experts should spend more time than the intermediates to reach decisions, especially given results such as those reported by J. R. Anderson (1980) in which he describes experts performing tasks faster than non-experts. However, many of the task domains described by J. R. Anderson, such as cigar rolling and flash arithmetic, while not simple, are less complex than design tasks. For more complex tasks, such as manufacturing, planning (Hayes & Wright, 1990), military planning (Marshak, 1999), and equine nutrition (R. Anderson, 2003), experts have been observed to take more time to complete problem solving tasks than non-experts. The explanation offered in military and manufacturing domains has been that non-experts are simply not completing as many problem solving steps or considering as many issues as experts. The non-experts lacked the experience to know they should be doing these steps or considering these options. Solution quality usually suffered as a result.

User preferences

A survey given to each subject showed that 63% of the experts preferred the fuzzy decision aid, 38% preferred the deterministic aid, and none preferred to use no aid. In contrast, 50% of the intermediates favored the deterministic decision aid, 39% the fuzzy aid, and 11% no aid.

It is interesting that the intermediate-level product designers preferred a decision aid over no aid because it is not clear that the aid provided significant benefits, and it required more time than doing the task by hand. More surprising still is that the experts preferred the fuzzy decision aid over the deterministic aid; both provided similar benefits in increased consistency in rankings (which may indicate increased decision quality) but because the fuzzy aid required more time one might expect them to prefer the deterministic aid.

While users do not always prefer the method that improves performance the most, it is important to understand what users' preferences are as indicators of what methods they may be willing to adopt and use, given the right conditions.

However, it is not uncommon for subjects in a laboratory experiment to express a preference for a technique that does not actually improve their performance (Morse et al., 1998). In this case, the subjects' preferences may have reflected an intellectual appreciation of the mathematical methods encoded in the decision aids. The intermediates may have preferred the deterministic method for its simplicity. This may also reflect students' lower level of comfort with statistical concepts of uncertainty. The expert designers, however, were more versed in and had a better working appreciation for the uncertainty associated with the cost and performance of design alternatives.

Ultimately, despite the preferences expressed during the laboratory study, subjects did not necessarily use the decision aids in subsequent product design work. An intellectual preference expressed during a laboratory experiment, in which subjects are removed from typical pressures and deadlines of the workplace, is not the same as a practical preference in the context of a working environment where perceived benefits must outweigh perceived costs. However, we believe it is important to understand users' preferences because it may impact the ease with which they are willing to accept and use a particular decision aid, given that an appropriate balance of costs and benefits can be achieved.

Previous | Next