upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Expert review

Fifteen information design experts were recruited to participate in a heuristic evaluation and ranking of the resulting information structures from the Modified-Delphi and Open card sorting studies. On average, the participants had 3 or more years of professional experience as an information expert and spent most of their time at work on information architecture-related activities. The study was conducted online via a browser-based form that provided instructions and a method for answering a series of questions. Participants were given background information on the users and goals of the website and a downloadable copy of the results of the Modified-Delphi and Open card sorting studies (Appendix 1, Appendix 2). They were then asked to provide a score from 1 (very poor) to 5 (very good) for a series of information architecture heuristics based on industry best practices and Rosenfeld's (2004) information architecture heuristics:

  1. Breadth and depth are balanced.
  2. Labels are clear and meaningful.
  3. Data is of similar granularity and dimension.
  4. Naming scheme is consistent and logical.
  5. Visual hierarchy is clear.
  6. Organization fits users' needs.

In addition to the heuristics, the participants were asked for their overall impression of the information structure and asked to rank the two information structures. This was a within subject study with all the participants rating both information structures from the Modified-Delphi and Open card sorting studies. The information structures were anonymized and counterbalanced to prevent bias. For half of the expert reviewers, the Modified-Delphi information structure was labeled as Information Structure A and presented first, and the Open information structure was labeled as Information Structure B and presented second. For the other half of the expert reviewers, the Open information structure was labeled as Information Structure A and presented first, and the Modified-Delphi information structure was labeled as Information Structure B and presented second. Participants were informed that the two information structures were generated from card sorting studies, but not that the card sorting studies employed different methodologies.

After both of the information structures were reviewed, the participants were asked to rank which structure was better, or if they were the same.

Inverse card sort

Seven participants were recruited for an Inverse card sort of the resulting information structures from the Modified-Delphi and Open card sorting studies. The recruiting was based on the same criteria as the card sorting studies: four current law students, two law professionals, and one undergraduate pre-law student. The study was conducted online via a browser-based form that provided instructions and a method for answering a series of questions. Participants were asked to select the category where they would expect to find the answer to the question. The questions were derived from the results of the exit questionnaire administered during the Modified-Delphi and Open card sorting studies. The categories were the top level categories from the information structures generated from the Modified-Delphi and Open card sorting studies.

The Inverse card sort was also a within subject study, with all the participants answering the same set of questions for the anonymized Modified-Delphi and Open information structures. The order in which the questions appeared was counterbalanced to prevent a learning bias. Half of the participants were asked to answer questions for the Modified-Delphi information structure first and the Open information structure second, and the other half of the participants were asked to answer questions in the reverse order.

Previous | Next