upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Card sorting is a participatory, user-centered design activity that information architects use to gain an understanding of how users understand and model information (Maurer et al., 2004). This method is used to draw out underlying mental models (Nielsen et al., 1995; Rosenfeld et al., 2002) that will later aid in the design or validation of an information architecture. A participant in a card sorting study is given a set of cards that each contains a piece of information. The participant sorts the cards into groups and labels each group. These results are then analyzed against a hypothesis by cluster analysis, by affinity mapping, or by simple pattern matching. Specifications on the methodology and what the researcher does with the results depend on the type of card sort conducted. Studies may be conducted in a laboratory setting with a stack of note cards and a table, on a computer in a laboratory (Toro, 2006; Classified, 2007), or over the Internet (OptimalSort, 2007; Socratic Technologies, 2007; WebCAT, 2007; WebSort, 2007).

There are a number of card sorting methods that have been used as research tools by psychologists and information designers; however, there are two types of card sorts that are used at different stages in the design of an information architecture: pre-design and post-design methods. Pre-design methods are used early in the design process to gather input for creating an information architecture. Post-design methods are used after an information architecture is developed to validate or edit an existing architecture.

The Open card sort is a pre-design method where participants sort cards into categories they create themselves. It is one of the earliest design methods information architects employ to aid in creating an information architecture. Participants have very few restrictions on how they can work with the cards; they can rename cards with better labels, add or remove cards from the final structure, or place the same card in multiple places. This freedom makes the method one of the strongest for drawing out an underlying mental model of the participants. A number of methods for analyzing the results of Open card sorts, with some of the more common metrics being cluster analysis-analyzing the relationship of a card to a category and the relationship of a card with another card (Rosenfeld et al., 2002; Toro, 2006; Tullis et al., 2004).

The Closed card sort is a post-design method where participants sort cards into preexisting categories. Participants do not have as much freedom with the cards as in an Open card sort and must use the categories and labels provided to them by the study administrator. This method can be used in two ways: to add new content to an existing information architecture or to test an information architecture by scoring participant results with the existing structure.

The Inverse card sort, also known as reverse card lookup, is another post-design method which is a variation of the Closed card sort. The top levels of the information architecture are provided to the participants, and they are asked to select where they would expect to find low-level content based on task- or topic-based scenarios (Classified, 2007; Scholtz et al., 1998). This is a useful method to quantitatively validate or rate an information architecture, similar to administering a quiz or test.

A card sort offers a number of benefits that make it attractive in the practice of information architecture (Maurer et al., 2004). It is widely practiced and effective for gathering user insights for an information architecture. Card sorting is also simple to conduct and relatively cheap, with the cost of materials being low compared to other user-centered design study methods. However, the method has several weaknesses, some of which negate the proposed benefits. First, the activity of organizing cards is out of the context of the user's goals and tasks. Organizing a set of data in a laboratory session is much different than wayfinding on a live website. Second, consistency between participants may vary, especially when a dataset has multiple possible organization schemes. Lack of consistency can weaken category and card relationship results. Open card sorting tries to solve this issue with larger numbers of participants to gather more data.

Previous | Next