upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


The goal of a participatory design method is to gain insight from users of a system, not the designers. Introducing a model influenced by the information architect transforms the method from an information gathering technique to an information validating technique. Using an information architect in the study also presents a social issue. An information architect-because of his or her professional status, knowledge of the information, relationship with the client, and relationship with the participant-may be held in higher regard than a peer, and knowledge of an information architect's influence may intimidate participants from altering the information structure model. Additionally, participants should never be told how many previous participants have worked on the information structure, because the number of participants may be intimidating and prevent a participant from being comfortable making changes.

Recruiting for the Modified-Delphi method is similar to recruiting for any other user-centered design study. The Delphi method is traditionally a method of expert opinion, and the users of a product could be considered an "expert" on the product. Approximately 8 to 10 participants have been the typical number of experts recruited in traditional Delphi studies. Depending on the goals or needs of the product design, you may recruit participants who are the target users and mixed user types, a single user group of particular interest, or the primary user group. However, if the participant types are very different and propose very different models to work with, this instability may prevent the study from reaching a consensus.

Other reasons for not reaching a consensus may be the existence of conflict cards. These are topic cards that do not stabilize in a category and participants cannot reach a consensus by the end of the study. There are a number of reasons why there may be conflict cards including:

Patterns of conflict may be identified during the study or during analysis. It is useful to look back at participants' comments from the study sessions to see if there were any indications as to why a topic card may have had issues. Conflict cards are not necessarily a bad thing; they identify weak points in the information dataset so the information architect can pay special attention to them when designing the information architecture.

Analysis can be done as with the Open card sort using affinity mapping or another pattern matching technique; however, special attention should be paid to the final participant's work. The final participant has had the influence of all the previous participants and should have had the fewest significant changes. In a model study, their work will be very similar to the final results analysis and can be used as a preliminary result or a metric to compare against the final analysis.

The goal of the Delphi method is to reach a consensus in a body of knowledge. In information architecture, there is rarely a single correct answer, but usually a few suitable answers that will accommodate most of the audience. If a proposed model is not agreeable to a participant, the participant is free to propose a new model by scooping up the cards and beginning from scratch. During a Modified-Delphi card sorting study, it is possible that many models are proposed and no consensus can be reached; in that case, variables such as homogeny of the dataset, participant experience with the topic, and participant makeup should be considered.

There may be several logical structures for the information, and it is up to the information architect to choose which one is the most appropriate. In this case, the Modified-Delphi card sort may be too relaxed an information gathering tool, and more directed studies after a model is selected may be more useful. The Modified-Delphi card sort is meant to be a more practical pre-design activity to aid in the design of an information architecture. Improving the quality of results from each participant, reducing the time to conduct the study and analyze the results, and lowering the costs of conducting a study and possibly cognitive costs to the participants are all potential and expected benefits.

Previous | Next