upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Summary of results

The results from the studies presented in this thesis provide compelling evidence that as a laboratory method, the Modified-Delphi card sort provides an alternative that is better than the Open card sort for gathering input for an information architecture early in the design process (see Table 9).

The independent information structure ranking suggests that information design experts thought both the Modified-Delphi and Open information structures would be useful for aiding in the design of an information architecture. However, the overall rating and the dependent information structure ranking both offer statistically significant evidence that information design experts thought the Modified-Delphi information structure was better than the Open information structure in terms of heuristics and more useful for aiding in the design of an information architecture. Although not every heuristic ranking was found statistically significant (3 out of 4 rankings were significant at the α = 0.5 level), the Modified-Delphi information structure was ranked higher than the Open information structure in all counts.

The results from the Inverse card sort were surprisingly very poor. Low results were expected because the tested information structures were not refined information architectures, however a performance of 46% was well below the expected score. Although only seven participants were tested, a number too small for statistical tests, this number of participants is typical to how a usability study would recruit. Interestingly, the information structures from the Modified-Delphi and Open studies performed similarly in most of the questions and high or low scores, even though the questions were selected because of the differences in the information structures. This suggests that the topic questions that had the poorest scores may be sensitive to the types of participants (who were overwhelmingly students) and must be given special consideration when refining the final information architecture.

Table 9. Summary of Results from Method Comparison Studies
Comparison of Sorting Method Heuristic Review Overall Rating Independent Ranking Dependent Ranking Inverse Card Sort
Modified-Delphi < Open          
Modified-Delphi = Open     X   X**
Modified-Delphi > Open X* X   X  

* not all heuristic scores were statistically significant at the α = 0.05 level
** not found to be statistically significant at the α = 0.05 level due to small sample size

Previous | Next