upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Creating Effective Decision Aids for Complex Tasks

Caroline Clarke Hayes and Farnaz Akhavi

Journal of Usability Studies, Volume 3, Issue 4, August 2008, pp. 152-172

Article Contents


Creating effective decision aids is not simply a matter of finding a method that computes the most correct answer or the interface that best presents the data, but also of finding the most effective way to integrate tools with human problem solving needs. For example, tools based on mathematically correct and sophisticated models may not actually improve problem solving performance if they frame the problem in a way that does not fit human problem solvers' approaches.

In this work, we have chosen to study decision aids that support the product design process, because the ability to compete in the global economy is highly dependant on the ability to rapidly produce high quality, low cost products. However, products such as cell phones, healthcare systems, or space stations are becoming increasingly complex. This means that product designers face greater decision making challenges than ever before.

We have further chosen to focus on design decisions made during the very early, conceptual stages of design, because decisions made at this stage have the largest impact on the cost, quality, and success of a product (Ishii, 2004). By the time one gets to the final stages of the design process, the major decisions have already been made and further choices have relatively little impact. Unfortunately, the conceptual design stage is an inconvenient time at which to make decisions and commitments because choices based on cost and performance must be made between alternatives that are little more than rough. It is simply not possible to produce accurate assessments of cost and performance during the conceptual design stage.

Yet product designers must make choices anyway in order to make the design task manageable. They must because there may be hundreds or even thousands of alternative designs and possible variants for a single complex product. It may require a team of 20, 100, or more designers to fully develop even one alternative to a sufficient level of detail that accurate cost and performance estimates can be produced. Thus, it is simply too expensive to develop them all to a level of detail sufficient to allow one to accurately choose the best with certainty. Some may argue that it is inappropriate to apply mathematical methods during conceptual design because of the uncertainty and lack of detail. However, this is where designers face the largest decision making challenges and where improvements can have the largest impact.

Mathematical decision making approaches that represent the uncertainty of a situation have long held great theoretical appeal for helping product designers make better design decisions for all the reasons above. However, while product designers routinely use many computer tools to help them visualize, analyze, and simulate the performance of products, mathematical decision methods are not used consistently in their daily work. This is not to say that they do not use them. In fact they do. Many companies use a variety of decision making techniques to explain or justify major decisions. However, designers do not tend to use these methods in their day-to-day design decision making to the extent that one might expect.

The overarching goals of this work are to begin to develop a better understanding of why mathematical decision methods have not been embraced by designers in the workplace, and how the mathematical decision methods can be made to better support designers' needs in the workplace. The more immediate goals of this paper are to quantify benefits and costs experienced by designers when using a variety of different mathematically-based decision aids in the laboratory, to better understand how product designers go about decision making in the workplace, and to use this understanding to explain the laboratory study results, and to inform future research directions.

This paper begins with a presentation of our model of the product design process that incorporates not only our own data and observations but also unifies a number of models created by other researchers to describe the design process. To study the issues above we used a combination of quantitative and qualitative methods that include the following studies:

The research questions that we viewed as relevant to ask evolved during the sequence of studies reported as we gained knowledge of product designers, their approaches to decision making, and the challenges of their work environments. Initially, we set about asking the question: "Which decision aid supports product designers better - one that allows them to express their uncertainty about the price and performance of the design alternatives, or one that does not" Given the inherent uncertainty and incomplete knowledge associated with conceptual designs, we assumed that the former would yield the best results.

However, much to our surprise, our experiments did not show this to be true. The decision aid that allowed product designers to express uncertainty did not yield significantly better or worse results than the other decision aid. Furthermore, while both decision aids produced better results under certain circumstances than no decision aid, it was not clear that their benefits necessarily outweighed their costs (time for data entry, software installation, software training, etc.). This situation was unexpected and was far more nuanced than we had initially envisioned.

At this point, our questions turned towards finding explanations and a deeper understanding: "Why didn't the 'uncertainty' decision aid yield more benefits than the other?" "Why didn't either decision aid yield a clearer balance of benefits?" "How well does the framework imposed by most mathematical decision approaches fit with product designers' actual approaches to decision making in the workplace?" and "How can we make tools that better support product designers' actual needs in the workplace?" These questions form a future research agenda for development of human-centered decision aids to meet the workplace needs of designers working in any complex and uncertain domain. We feel that the results of such a research agenda will apply to electro-mechanical product design and, more generally, to any type of complex design task.

Previous | Next