upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

The Usability of Computerized Card Sorting: A Comparison of Three Applications by Researchers and End Users

Barbara S. Chaparro, Veronica D. Hinkle, and Shannon K. Riley

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 31-48

Article Contents


Card sorting is a popular technique used in the areas of information architecture, cognitive psychology, and cognitive anthropology to capture mental models of how participants organize information. Card sorting is also used to help define a website structure or software application menu structure because it explores how users conceptually organize information. To conduct a card sort, participants are given a stack of index cards each containing an informational item to be sorted. They are asked to group the cards into categories that make sense to them and to name the groups they create. They may then refine their groups by subdividing any large groups into smaller subgroups or combining small groups. Participants work individually or in small groups to sort the information. This grouping data is gathered across many participants and summarized using either cluster analysis techniques or frequency analyses of item groupings. These results are then used to generate an overall structure of the information. For software and website content, the results also lead to suggestions for navigation, menus, and possible taxonomies (see Courage and Baxter, 2005, for a detailed overview of card sorting).

While card sorting has traditionally been conducted using physical cards, computer and web-based card sorting (hereafter, referred to as electronic card sorting) applications are becoming a popular alternative. Electronic card sorting has several advantages over physical card sorting, which include the following advantages (Zavod, Rickert, & Brown, 2002):

No significant differences have been demonstrated between manual and electronic card sorts in terms of accuracy, test-retest reliability, and number of categories generated by participants (Harper & Van Duyne, 2002) or for closed card sort results (Bussolon, Russi, & Del Missier, 2006). However, electronic card sorts with first-time users have been demonstrated to take longer than manual card sorts, although this time decreases with subsequent uses of the application (Harper & Van Duyne, 2002).

In spite of its advantages, one important aspect of electronic card sorting that has not received much attention in the literature is the usability of the card sort applications themselves. There are two primary populations involved in the use of a card sort application: the researchers, who organize the card sort and analyze the data; and the end users, who participate in the actual card sort exercise. Each user group has different needs though usability is very important to each. Researchers want a program that allows quick creation of card sets and easy access to data for analysis and reporting. End users need a program that provides an intuitive interface, easy manipulation of cards, and a simple way to name the card groups. Given that an electronic card sort activity can be widely distributed via the Internet, it is possible that the researcher and the end user may never interact face-to-face. Therefore, the intuitiveness of the program is extremely important to insure quality data collection.

There are many electronic card sorting applications available to practitioners today. A summary of these programs are provided in Table 1.

Table 1. Electronic Card Sort Applications.
Program (URL) Platform Availability
Windows Commercial product
Web-based Commercial service
Web-based Commercial service
Web-based Commercial service
Windows Commercial product
MAC OS X Commercial product
Windows UNIX Free
USort/EZCalc (IBM)1
Windows Free but now archived, no longer supported

The purpose of this study was to examine the usability of electronic card sorting programs from the perspective of the researcher and the end-user. A two-part study investigating the first-time usage of three electronic card sort programs for an open card sort was conducted. Study 1 examined the usability of the applications from the researcher perspective, and Study 2 examined the usability of the applications from the end user perspective.

The three card sorting applications evaluated were the following:

Previous | Next