upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


In the real world, the results of a pre-design card sort should not be converted to an information architecture or be used to verify the final information architecture without the influence or input from an information design expert. However, for the purpose of comparing two pre-design methods without the bias and experience of an information design expert, this was the best way obtain accurate results for this study. Pre-design card sorting methods are designed to gather insights on how users model information, but do not take in to account user tasks, context of use, or client goals. They offer a useful way to gain design insight to aid in the design of an information architecture and involve users in the design process.

During both of the studies, I observed an interesting behavioral difference between participants of the Modified-Delphi card sorting study and the Open cart sorting study. Most of the Modified-Delphi study participants were very talkative during their sessions, asking questions about cards and the instructions, talking about the decisions they were making as they organized the cards, their experiences they have had on law school websites, or about their lives in general. Participants of the Open study were quite the opposite, with only a few participants asking questions about the cards or the instructions, and no one talking aloud during their process or sharing any experiential or personal information. Although this behavior is reported anecdotally, this may be an indication of how high the cognitive load is for the Open card sort in comparison to the Modified-Delphi card sort. Each Open card sort participant must review all of the cards, create a model, and then refine that model. The Modified-Delphi participants had help with the creation of a model by only having to make modifications to other participants' work that they review. The cards were presented to them in a logical order, and participants simply had to form an opinion about the model and modify it, rather than creating a model from scratch and refining it. Besides being more participant-friendly, the possibility of a lower cognitive load may make the Modified-Delphi method appropriate for studying larger or more difficult datasets that are usually avoided in Open card sorting because of the cognitive costs to participants.

During the Modified-Delphi card sorting study, three of the participants (not including the seed participant) decided to start from scratch and scooped up all the cards in a pile before beginning their work, destroying the previous participants' work. This was expected to create a noticeable change in the evolving structure; however, during analysis, there was still a great similarity between the participants who based their work off of others and the participants who started from scratch. During the study instructions, participants were asked to review the provided structure before beginning their work. This priming (Tulving, 1990) of the previously proposed model may have influenced them on some scale. Their comments indicated that they were already familiar with information in the dataset and felt more comfortable organizing it on their own terms.

During the analysis phase of the parallel card sorting studies, there was a noticeable difference in the difficulty analyzing results between the two methods. It took more than twice as long to analyze the Open card sort results than the Modified-Delphi card sort. In addition, there were two additional participants in the Open card sorting study than the Modified-Delphi study, but contrary to expectation, it was not during the card entry in to the spreadsheet where the time was lost. This difference in analysis time may be attributed to the Modified-Delphi card sorting study results having a much higher agreement weight than the Open card sorting study. Seventy-three percent of the cards in the Modified-Delphi card sorting study had a greater than 50% agreement weight that made the decision on where to place card in the final information structure category very quick and easy. However, only 21% of the cards in the Open card sorting study had greater than 50% agreement weight. This meant that for the remaining 71 cards, at least one of the placement heuristics described in the Methodology section had to be employed. This process was very time consuming and essentially made the analysis of the Open card sorting study more expensive.

The two studies had very similar user group combinations, which strengthens the comparison between the two methods (see Table 8). The Modified-Delphi method did have a disproportionate number of women over men, but I do not think this had an impact on the study. The Inverse card sort had a small number of participants (seven); however, this number is typical in most user-centered design studies. There were also a disproportionate number of students (four current law students, one undergraduate pre-law out of seven participants) who participated compared to the other study participant makeups. This may have had an effect on the Inverse card sort study results.

Table 8. Summary of Study Participant Groups
User Group Modified-Delphi Card Sort Open Card Sort
Undergraduate Pre-Law Students 2 1
Current Law Students 2 3
Law School Faculty 1 2
Law School Administration 1 1
Law-Related Professional (non-Attorney) 1 2
Law-Related Professionals (Attorney) 1 1
Total number of participants 8 10

Previous | Next