upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Card Sorting: Current Practices and Beyond

Jed Wood and Larry Wood

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 1-6

Article Contents


Number of Participants

Current practice:

Because of the similarity between card sorting and survey research, it is assumed by many researchers that the more participants that can be recruited for the card sorting task, the more valid the results will be. However, in addition to the increased effort and the logistical issues that might be involved in recruiting a large number of participants, a much greater effort is required to analyze a large body of data. Interestingly, the one study that has been published (Tullis & Wood, 2004) on this question indicates that useful results can be obtained with fewer participants than one might otherwise suppose.

Recommendation:

As few as 25-30 participants will likely yield results similar to those of several hundred, provided these participants are representative of actual users and are familiar with the domain being considered. Thus, researchers could save time and money, not to mention the headache of analyzing all the extra data.

Participant Instructions

Current practice:

When psychologists began to use card sorting as a methodology to study conceptual and category structures, they were careful to give non-directive instructions (e.g., "There is no right set of categories. Just group things together that seem to be similar in some way.") because they were concerned about biasing participants. Unfortunately, some information technology researchers have carried that practice over into current card sorting work.

Recommendation:

Avoiding bias is generally a noble goal, but this seems to counter the purpose of gathering information that is relevant to the goal of making it easier for users to navigate through Web pages. Hence, researchers should be explicit about the intended purpose of conducting the card sorting study (e.g., "Please group these items as you would expect to find them on our corporate intranet").

Participants' Complete Understanding of Items

Current practice:

Far too many information technology researchers are overly optimistic about their participants' familiarity with the items.

Recommendation:

Because it is impossible to guarantee that all participants will have a common understanding of the domain items, it is helpful to provide more detailed descriptions of the items to be sorted. In early research by psychologists, it was a common practice to include the description of an item on the back of the item card. Then participants could simply turn the card over and review the description to make certain they were thinking about the item in the intended manner. We strongly recommend that item descriptions be provided, even in the most promising situations (e.g., a corporate intranet with employees as participants).

Open vs. Closed Sorting Tasks

Current practice:

Most card sorting projects are open sort, where participants are given a list of content items representative of the content that is being planned (or already exists) for some portion of a Web site. The participants are then asked to categorize the items in a way that represents their best organization for the Web site. Occasionally, researchers have data or experience that appears to justify the use of an existing organization (i.e., a pre-existing set of useful categories) into which participants could be invited to sort the items, resulting in what is termed as a closed card sorting project. The assumption seems to be that the existing structure is close to optimal-it just needs some minor adjustment.

Too frequently in our experience, researchers are convinced they have a list of categories that is useful, but just needs some minor adjustment. However, they have little or no evidence related to whether or not potential users are of the same mind. In the extreme, we know of a project in which users refused to participate because they experienced such difficulty in making meaningful sense of the categories in which they were asked to sort a set of content items.

Recommendation:

Begin research with an open sort. Only after carefully analyzing the data should a closed sort be conducted for validation. If the decision is made to begin with a closed sort, make it simple and be prepared to obtain less-than-optimal results.

Previous | Next