upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Creating Effective Decision Aids for Complex Tasks

Caroline Clarke Hayes and Farnaz Akhavi

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 152-172

Article Contents


Discussion

The following sections discuss some questions that occurred as a result of the study.

What may discourage use of mathematical decision aids?

First, it was simply too time consuming to use mathematical methods for all decisions. As mentioned earlier, if these methods were applied to every small design variation, the design process would become exceedingly slow without improvement in most decisions, especially when there is one obvious winner. Users of any tool (software or otherwise) are very sensitive to the costs and benefits that they personally derive from the tools, and they may not be willing to use them if they perceive the benefits to be smaller on average than the extra work required (Grudin, 1988).

Second, designers did not have a clear metric or rule of thumb that allowed them to identify situations in which the mathematical tools would be likely to yield benefits. When designers are faced with a situation in which they are not sure whether time consuming mathematical methods will provide benefits, it is only natural to chose not to put in the additional work required to use them.

Third, the mathematical methods did not allow designers the flexibility to which they were accustomed when comparing design alternatives. We observed that designers would often incrementally consider design criteria, starting with those they considered most important, and conditionally exploring less important criteria as "tie breakers" if a winner did not emerge (this is the flexible considered comparison strategy which we described earlier). Designers can save much time by only considering criteria when they need to and only considering them for specific alternatives. In contrast, the mathematical methods assume that a fixed set of criteria will be compared and users must specify all of them, regardless.

Finally, and probably most importantly, most mathematical models assume a rather limited view of decision making (Klein, 1993). Naturalistic decision making is an approach in which decision making is studied in the context of actual tasks and the environments in which they are typically carried out (often work environments) (Klein, 1993; Suchman, 1987). Some of the premises underlying research in naturalistic decision making (Orasanu & Connolly, 1993) are that traditional, mathematically-oriented decision making research focuses on only one part of decision making, the decision event. In a decision event, a single decision maker compares a fixed set of alternatives using a fixed and well defined set of goals. Additionally, if precise information on the performance of each alternative is not available, statistical estimates can be obtained.

However, in natural design decision making situations (Dym, 1994; Thomas & Carroll, 1984; Ullman et al., 1988) few of these assumptions hold. Design is a good example of such a task; alternatives are constantly being added, modified, or refined, as are design goals. Much information is simply unknown or hard to obtain. Additionally, there are practical time considerations; the sheer number of decisions means that most must be made very rapidly. Finally, many important design activities are not captured by a decision event, for example, the information seeking behaviors that precede the selection of an alternative.

Why didn’t designers benefit more from the non-deterministic method?

One might expect that the non-deterministic (fuzzy) method would produce better results than the deterministic method because the former are based on more information. However, this was not observed to be the case in the experiment and set-up described; the average "goodness" values produced for each alternative by the fuzzy and the deterministic methods were very similar to each other (Akhavi, 2006). Thus non-deterministic methods may not provide any direct benefit if applied only to the task of ranking alternatives in a classically framed decision event. Even if the uncertainty of each alternative's value is displayed, it may not produce a significantly better ranking.

Thus, by framing design decisions as "decision events," one may be asking the wrong question, "Which alternative is best?" or more accurately, not enough questions. In practice, designers intertwine the questions of "Which alternative is best?" with "Do I have enough information to decide which is best?" The uncertainty in the values of the alternatives can be directly used to assist designers in answering this second question, as illustrated in the example in the next section. Thus, by applying non-deterministic MCDM methods to a wider range of tasks (alternative selection and information seeking decisions) they may better support a designer's practical needs.

Previous | Next