upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


The question of how many participants to include in a card sorting study is under debate, particularly with regard to Open card sorting. Some card sorting guides suggest as few as four to six participants (Gaffney, 2000; Robertson, 2002), others suggest 10 to 15 participants (Maurer et al., 2004; Nielsen, 2004), while others suggest as many as 20 or more (McGovern, 2002; Tullis et al., 2004). Tullis and Wood (2004) have noted a minimum number of 20 to 30 participants are necessary to get meaningful numbers from an Open card sorting study. More participants may help provide more consistent results, but the larger sample size also increases costs and analysis time. In practice, even 10 to 15 participants is a high number of participants for a study. Informal polls conducted at the 2007 Information Architecture Summit and local Usability Professionals Association meetings revealed that many practitioners were conducting card sorting studies with 6 to 12 participants.

Using 6 to 12 participants is practical for practitioners for a number of reasons. A large number of participants mean higher costs of additional participant stipends, moderator fees, facility costs, and analysis hours. For companies who do not have their own facilities, the cost of renting a lab is a significant part of the study costs. Limiting the number of participants in a study to a number that can be scheduled in one day could reduce some of these costs. However, without conducting an Open card sorting study with an adequate number of participants, results from the study may not be reliable or provide the quality of input necessary for designing an information architecture.

A strong need exists for a more reliable and less expensive card sorting method that information architects can use early in the design process. This method must have strengths in three areas: results, time, and cost. It must be easy to conduct and not require many participants or a long study period. It must provide results that are both useful and worth the amount of time and money necessary to collect them. It must be low in cost so it can be easily funded and justified.

As previously mentioned, there are a number of web-based card sorting tools and services available to conduct an online card sort. OptimalSort (2007) supports closed card sorting. WebCat (2007), Soctratic Online Card Sort (Socratic Technologies, 2007), and WebSort (2007) support both open and close card sorting. Web-based card sorting has alleviated some of the expense of conducting an open card sort by turning to the web instead of a laboratory. Because participants can participate online instead of commuting to the testing facility, there are no facility rental fees and participation stipends can be lower. This cost savings can be invested in recruiting more participants to reach the 20 to 30 participant range recommended by Tullis and Wood (2004). However, the quality of the results gathered from an Open card sort is still in question.

The goal of this research was to develop a new card sorting method that provides better results and overcomes the previously discussed weaknesses of Open card sorting. I propose a new methodology similar to the Open card sort that is based on a forecasting technique called the Delphi method. The Delphi method is a moderation process for allowing multiple participants to work together towards a solution, while minimizing collaboration bias. Some of these biases include the bandwagon effect, which is the believing in a certain position because others do (Nadeau et al., 1993); herd behavior, which is a defense mechanism that results in following of the crowd (Hamilton, 1971); or the dominance of a strong personality on a group (Boy, 1997). Instead of producing a number of models created by individual participants that are then combined, averaged, and analyzed to draw a final conclusion, this new method allows participants to work individually on a single model until that model can be accepted without major additional modifications. While Open card sorting records multiple mental models and tries to draw conclusions from the results by analyzing statistical characteristics, Modified-Delphi card sorting proposes a single mental model that is then modified until a consensus is met.

Previous | Next