upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


When I proposed the Modified-Delphi card sort, there were three goals I wanted to meet in order to have successfully proposed a pre-design method to replace the Open card sort: (a) improve the quality of results from each participant, (b) reduce the time to conduct the study and analyze the results, (c) lower the costs of conducting a study and possibly the cognitive costs to participants.

Results. There is a proven benefit of participants working with a single model, rather than many. The information evolutionary model influenced by the Delphi method helps control randomness and outliers that are commonly encountered when analyzing multiple models as a single set of data. Results from the studies conducted also suggest the superiority of results generated from the Modified-Delphi card sort over the Open card sort.

Time. By reducing the recommended number of participants to complete a successful study, the amount of time required to both conduct a pre-design card sorting study and analyze the results has been reduced. The amount of time saved during analysis may be even greater due to the quality of results gathered, as discussed in the Discussion section. The weak agreement between the Open card sorting results required extra analysis time to follow heuristics for card placement, thus increasing the difficulty in analysis and the time for analysis.

Costs. Time is money (Walker et al., 2003), and by reducing the overall time of a study, the costs are also reduced. Combined with fewer participant stipends and fewer days of facility costs, the overall return on investment of the Modified-Delphi card sort is significantly higher over the Open card sort. While not monetary in nature, there is also a savings of cognitive costs to the participants. As discussed previously, participants of the Modified-Delphi card sorting study were much more talkative than the participants of the Open card sorting study. It is possible that the method reduced cognitive cost and helped the participant to be more engaged in the problem solving parts of the task.

There is strong statistically significant evidence that as a laboratory method, the Modified-Delphi card sorting method is better than the Open card sorting method. Results from the heuristic review and ranking by information experts show the overall rating of the results from the Modified-Delphi card sort were better than the results from the Open card sort. The information structure generated from the Modified-Delphi card sort was also considered to be more helpful for aiding in the design of an information architecture than the information structure generated from the Open card sort. Also, many of the expert heuristic review scores from the Modified-Delphi card sort were significantly better than the results from the Open card sort; those that were not statistically significant still provide evidence that the two methods are at least equivalent.

The Modified-Delphi method is a new card sorting method that requires fewer users per study and provides better results than the traditional Open card sort. Additionally, results from the study suggest a savings of time and costs from using the Modified-Delphi card sort over the Open card sort. A parallel study of the two methods has provided statistically significant evidence that results from the Modified-Delphi card sort are at least as good as the results from the Open card sort, and in some cases, better.

Previous | Next