upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

The Usability of Computerized Card Sorting: A Comparison of Three Applications by Researchers and End Users

Barbara S. Chaparro, Veronica D. Hinkle, and Shannon K. Riley

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 31-48

Article Contents


Results from this two-part study are interesting in that the “best” electronic card sort application appears to be dependent on the participant group using it. Researchers preferred WebSort to set up and analyze an open card sort while end users preferred OpenSort to do the actual card sorting. It should be emphasized that this study examined performance upon first-time usage. It is expected that with continued use, all participants would become more comfortable with the programs and find them easier to use. However, the process of a card sort activity requires only a single participation by each end user. Therefore, first-time perceptions of usability are extremely important to this user group. One may argue that researchers may spend more time experimenting with the application before using it and therefore, first-time usability may be less important. Still, first impressions are a critical factor in the face validity of a product and subsequent decision-making of whether to use it again.

The disparate results of this study underscore the importance of usability testing during product design for all user groups. It is possible that the application developers of OpenSort focused more on the end user experience than the researcher experience. Likewise, the developers of WebSort may have focused more on facilitating researcher activities rather than the end user experience. Electronic card sorting has great potential to expedite the process and improve the generalizability of its results by involving users from remote locations. However, these benefits will only be realized if the card sorting applications themselves are intuitive, efficient, and well-liked by researchers and end users.

Table 5 provides a feature summary of the programs evaluated in this study. As noted earlier in Table 1, there are many other card sorting programs also available. Practitioners are encouraged to evaluate these programs and see how they compare to those evaluated in this study in terms of usability and satisfaction, both from the researcher and end user perspective.

Table 5. Summary of Features by Program.
Feature CardSort WebSort OpenSort
Allows import and paste of text list of card items   X X
Allows images to be placed on cards   X X
Online help available for researcher X X X
Allows researcher to customize sorting instructions   X X
Provides ability to download raw data   X X
Method of displaying cards Stacked cards Vertical list of terms Stacked Cards
Allows users to name groups as they sort   X X
Allows users to create duplicate cards     X
Allows users to create subgroups     X
Provides online help for end user while sorting   X X
Generates dendrogram as results X X X
Provides ability to manipulate dendrogram     X
Provides additional methods of visualizing results     X


We thank Rashmi Sinha from MindCanvas for graciously providing free access to OpenSort for this study.

Previous | Next