upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


The following sections present information about information structures, the exit questionnaire, the heuristic review results, independent information structure ranking, dependent information structure ranking, and the Inverse card sort results.

Information structures

The information structures generated were based on guidelines specified in the Methodology section. Guidelines for the Modified-Delphi and Open card sorting studies can be found in Appendix 1 and Appendix 2, respectively.

In the results from the Modified-Delphi card sorting study, 66 out of 90 of the original set of cards had a greater than 50% of agreement weight (see Table 1). More than half of the participants agreed on the location of 73% of the cards. Also, nine out of the final 10 categories were represented in these cards. The final participant's work was also very similar to the combined information structure. Eight out of the final 10 categories were represented by participant's raw results; an additional category from this participant had been merged with another that made 9 out of 10 in the information structure. Seven cards did not match with the final information structure; they were "floating" cards of special interest and were not directly organized. Fewer than 90 of the original cards were represented in the final participant's work. There were a total of 75 cards represented in their structure including several "grouped" cards (bound by paper clips) that were counted as a single card and nine new cards were added through the study. In the results from the Open card sorting study, 19 out of 90 of the original set of cards had a greater than 50% agreement weight. More than half of the participants agreed on the location of 21% of the cards. Eight of the final 11 categories were represented in these cards.

Although the number of categories represented by the high agreement cards in the Open card sorting study is very close to the number represented by the Modified-Delphi card sorting study, there is a much greater difference between the two studies in the number of high agreement cards in general. The Modified-Delphi card sort provided 47 more high agreement cards (with a total of 66 out of 90 cards) than the Open card sort (with a total of 19 out of 90 cards).

Table 1. Summary of Agreement Weight Comparisons
Comparison Modified-Delphi Open
Number of cards with > 50% agreement weight 66/90 (73%) 19/90 (21%)
Number of categories represented by cards with > 50% agreement weight 9/10 8/11

Exit questionnaire

An exit questionnaire was administered to the 8 Modified-Delphi card sort participants and 10 Open card sort participants for a total of 18 completed questionnaires. Participants were asked to select what they felt were 10 of the most important cards to them. Their selections were based on their perspective of their individual audience needs, but because of the diversity of participants involved in the two card sorting studies, the selections were deemed to provide insight as to the most important topics of the dataset. Results of this questionnaire also helped determine questions for the Inverse card sort (see Table 2).

Fifty-seven out of 90 cards were selected by at least one participant with a median number of votes of 2 (excludes 0 vote cards). Given that half of the participants were current law students or undergraduate students interested in law school, it follows that the top selected cards relate to topics students are interested in: course listings and schedules, costs and ways to pay for school, and finding a job.

Table 2. Top 15 Responses to the Exit Questionnaire
Card Title Number of Votes
(18 Participants)
1. Catalog/Course Listings 9
2. Tuition 8
3. Career Services 7
4. Financial Aid Information 7
5. About the School 7
6. Academic Calendar 7
7. Scholarships & Loans 6
8. How to Apply 6
9. Academic Requirements 6
10. Course Descriptions 5
11. Library Services 5
12. Centers & Programs 5
13. Admissions Process 5
14. Faculty Profiles 5
15. University Facilities & Services 5

Previous | Next