upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Modified-Delphi and Open card sorting studies

Eighteen participants of the University of Baltimore School of Law website's target user groups were recruited to participate in one of two card sorting studies. These user groups included: law students, both current law students and undergraduate students interested in law school; law school staff, which included administration support staff, law professors, and other faculty members; law professionals, both attorneys who may or may not be university alumni and support specialists such as paralegals, records managers, and so on. Students and staff are the most frequent users of the website, but it was important to support professionals seeking information about clinics and seminars, especially alumni who are interested in donating to the school. The Modified-Delphi card sort was conducted with eight participants: one undergraduate interested in law school, two current law school students, one school administrator, one faculty member, two law professionals, and one attorney. The Open card sort was conducted with 10 participants: one undergraduate interested in law school, three current law school students, one school administrator, two faculty members, two law professionals, and one attorney.

Sessions for both the Modified-Delphi and Open card sorting studies lasted no longer than 60 minutes per participant. Before the session began, all participants were asked to complete a study consent form and answer additional background information questions about their experience using law school websites.

Participants of both studies were given a large table with the cards, a pen, and extra cards to use for naming groups, renaming card titles, or adding missing content. All of the Open card sort participants and Participant 1 of the Modified-Delphi card sorting study were given a set of 90 index cards containing high level topics from the current University of Baltimore School of Law website (http://law.ubalt.edu/). Participants 2 through 8 of the Modified-Delphi study were given the previous participant's results to work with. Participants of both studies were asked to create an organization for the content provided that made the most sense to them. They were permitted to change labels, remove cards that did not seem to fit, and add missing information. It should be noted that 90 cards were not enough to represent all of the content in the website and comments about missing content were expected.

There were two major differences in the instructions given to the participants of the Modified-Delphi card sort versus the Open card sort:

After the participants of both studies were satisfied with their work and declared themselves to be finished, a review of their work was conducted to clarify grouping and labeling decisions. A final questionnaire was administered that asked participants to select 10 of the most important topics from the original set of 90 information cards. The results of the questionnaire helped formulate questions for the Inverse card sort and to ensure that the possibility that unfamiliar cards were presented to participants of the Modified-Delphi card sorting study did not affect the final comparison of the two methods. Results of each participant session were recorded by taking a series of digital photographs of the participant's work so they could be recorded and analyzed at a later time.

Previous | Next