upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Current and Related Work

In Wisdom of Crowds (2004), Surowiecki discusses how, when accessed collectively, the masses have an intelligence that is rival to none. Four elements are necessary to form a "wise crowd": diversity of opinion, independence from other people's opinion, decentralization of knowledge, and a method for gathering the crowd. As an example of the power of collective knowledge, he discusses several kinds of markets, including prediction markets that rely strongly on expressing a position, rather than selecting a choice. According to Wikipedia (2007), "A prediction market would ask a question such as, 'Who do you think will win the election?' where an opinion poll would ask more specifically, 'Who will you vote for?'"

Similar to prediction markets, the Delphi method is a forecasting technique used to collect the opinions of a group of people in an objective way. It was developed by the RAND Corporation in the 1950's and 1960's (Helmer-Hirschberg, 1967; Linstone, 1975) as a way of gathering a knowledge base of military intelligence and experience without the influence of politics, rank, or other bias. It has since been applied to other domains such as technology, population sciences, and business.

The Delphi method is a technique that controls information gathered from participants through the study moderator. There are three key elements to the Delphi method: structure of information flow, feedback to the participants, and anonymity of the participants. Participants are often knowledgeable experts in a field and may have a personal stake in the resulting knowledge base generated from the study. This technique is similar to the Hegelian Principle of thesis, antithesis, and synthesis, where an argument and counter-argument are proposed and discussed until a consensus is reached (Stuter, 1996).

During the Delphi method, each participant was given the following information:

Figure 1. Hegelian Principle (the Delphi method)

Figure 1. Hegelian Principle (the Delphi method)

The moderator combines the previous session results and presents the material to the participants, careful to filter any bias or include personal opinion or experience that may be important. This direct interaction of the moderator is an important communication channel between the participants; however, it has also been noted as one of the weaknesses in the protocol (Teijlingen et al., 2005). Because the moderator has control over the collection and combination of information, they should maintain an objective view of the information and remain neutral on any presented positions. Conflict of interest may arise due to possible personal or business gains based on the results. To avoid this potential problem, unbiased, third-party moderators may be used for critical evaluations. Independent verification and validation is one such method used in the development of mission critical systems (Whitmore et al., 1997).

The Delphi method has benefits over other group communication methods. Instead of direct collaboration, participants interact with each other through the artifacts they leave at the end of their study session. An individual participant may not have the final answer, but a piece of what they have proposed may be valuable to the next participant. The opinions of others can be influential, valuable, insightful, or useless in a certain context. This anonymous collaboration alleviates peer pressure and other performance anxieties that are common to group collaboration methods and allows participants to focus on the problem.

Previous | Next