upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Use of Card Sorting for Online Course Site Organization Within an Integrated Science Curriculum

Alison Doubleday

Journal of Usability Studies, Volume 8, Issue 2, February 2013, pp. 41 - 54

Article Contents


The following sections discuss the study’s design, participants, materials used, and the procedures. This study involves two distinct card sorts, separated temporally by one year. Data from the first card sort were used to develop an initial course site, and data from the second card sort were used to modify the site.

Study Design

Card sorting and scenario-based usability tests were used to determine the organization of first-year course sites within the course management system. This study was conducted with approval from the UIC Institutional Review Board (UIC#2011-0359). The methodology of the current study can be summarized and subdivided into five specific stages:

A process flow diagram of these activities is presented in Figure 1.

Figure 1

Figure 1. Process flow diagram demonstrating stages of the study


There were two sets of participants who conducted card sorts: one set for the initial design of the site and one set for the modification of the design one year later. Participants from two first-year dental student cohorts at UIC College of Dentistry were recruited by email for participation in the card sorting activities. Course sites needed to be finalized and ready to use prior to the first day of the new curriculum. Unfortunately, the faculty did not have access to incoming students prior to this date so initial testing was conducted using students who had just completed their first year of the traditional curriculum (Cohort A). Although these students did not have experience with the new course structure, they had participated in many of the same laboratory and clinical activities, and they more closely resembled the incoming students in terms of demographics, background, and experience than the faculty or any other potential participant group. Data from the first set of participants, Cohort A, provided input for the initial course site design.

The curriculum planning committee decided that a second card sort with participants from the first class to actually experience the new curriculum for a year, Cohort B, would provide data for the modification of the initial design.

Cohort A participants, who had completed the first year of the traditional lecture–based, discipline-based curriculum, consisted of 17 students (6 males, 11 females, average age = 24.1). Cohort B participants, who had completed one year of the case-based, integrated, systems-based curriculum, consisted of 15 students (5 males, 10 females, average age = 24.3). Five additional students from each cohort were recruited to participate in a scenario-based usability test of each site design (Cohort A: n = 5, 1 male, 4 females; Cohort B: 2 males, 3 females). Participation in the study was voluntary, and no compensation was offered.


The free card sorting application for Mac operating systems, xSort (Enough Pepper, 2012 http://www.xsortapp.com/) was used in this study. The xSort application allows researchers to set up the sorting activity. Within the application, a simulated tabletop and stacks of cards allow for open, closed, and semi-closed card sorting. Researchers input card names, and participants use a computer mouse to move cards into groups (Figure 2).

Figure 2

Figure 2. Card sort within the program xSort (figure use and adaptation permission from xSort, http://www.xsortapp.com/): Participants group information together and can create hierarchies within the information architecture.

Regardless of whether an open, closed, or semi-closed sort is designated, xSort allows participants to leave some cards unsorted if they are unsure of how to group them. Any unsorted cards are excluded from the analysis.

Faculty members on the curriculum planning committee worked together to identify card names for the open card sorting activity. Card names were based on materials typically included in Blackboard course sites for first year dental students at UIC. The existing literature recommended using no fewer than 30 and no greater than 100 cards in order to make the most effective use of participants’ time (Spencer, 2009). Following those guidelines, faculty developed 45 cards, each representing one type of content typically found in a course site for first year dental students. The committee reviewed card names and discussed each in detail to ensure that each card represented a single type of content and that there was minimal or no overlap among card names.

One challenge of card sorting is avoiding keyword bias, or card names that might influence the sorting. For example, having a set of cards that included “anatomy videos,” “histology videos,” and “physiology videos” might bias a participant towards grouping them together into a “videos” category purely based on the card names, rather than on concepts that may be more important to the participant. In this study, card names were selected to minimize keyword bias as much as possible, and the author changed the word order or used synonyms to represent content whose title closely resembled the title of other cards. Descriptions were used as often as possible so “anatomy videos” might change to “recordings of laboratory dissections,” for example. Card names were then entered into xSort.


The following sections discuss specific procedures for each stage of the study.

Stage 1: Open card sort with Cohort A

The 17 participants from Cohort A were recruited by email. Summary statistics regarding self-reported gender and age of participants, in addition to the number of groups formed by each participant and the time for completion of the card sort, were collected in xSort.

During the open card-sorting activity Cohort A participants grouped related cards into categories and subsequently named each category. Participants had the ability to form as few or as many groupings as they desired and also had the option of removing cards from the sort. Within this open card sort, hierarchies were unrestricted and participants were able to group cards into subcategories. The author observed the participants and asked them to think-aloud, talking through the processes involved in grouping cards and naming groups. After sorting all cards, participants ranked the groups in order of importance. A faculty member took notes to record all observations, and each participant’s movements through the card-sorting program were captured with the screen-capture software Camtasia Studio (http://www.techsmith.com/camtasia.html).

Stage 2: Course site development and scenario-based usability test

Using the categories and information architecture gleaned from Cohort A’s open card sort, the author, with assistance from the Office of Dental Education at UIC, constructed a template course site in Blackboard.

Five additional participants from Cohort A, who had not participated in the open card sort, were recruited via email to test the template site. Participants were asked to complete nine task scenarios related to navigation through the site as shown in Table 1.

Table 1. Task Scenarios for Usability Test with Cohort A and Cohort B

Table 1

Stage 3: Implementation of the site in the new curriculum

The author (with assistance from the UIC Office of Dental Education) constructed all biomedical sciences course sites for the 2011-2012 academic year using the template site and the feedback from participants in the usability test. The new first-year class (Cohort B) used the newly constructed sites throughout the 2011-2012 academic year. Cohort B students also received a brief orientation demonstrating how to navigate the sites at the beginning of the Fall 2011 semester.

Stage 4: Semi-closed card sort with Cohort B

At the end of the 2011-2012 school year, 15 participants from Cohort B were recruited via email to participate in a semi-closed card sort. The purpose of this second card sort, separated temporally from the first card sort by one year, was to determine what modifications should be made to the course sites to make them more consistent with user needs. The first card sort was necessary to provide a starting point so that the course sites could exist prior to the start date of the new curriculum. The second card sort provided an opportunity to see if our conclusions about the organization and our predictions about what would make sense to students in the new curriculum (all based on data from the first card sort) were supported or not.

Cohort B participants were asked to group the same cards as in Cohort A’s open card sort. In the semi-closed card sort, however, Cohort B participants were provided with the names of the consensus categories appearing on the course sites and were asked to place cards within the named groups. Cohort B participants were able to add new groups or delete or rename existing groups. As with the Cohort A card sort, after sorting all cards, participants ranked groups in order of importance. All participants were observed and all movements within the card sort were captured using Camtasia Studio. Data from the semi-closed card sort were subjected to the same treatment as the open card sort data.

Stage 5: Course site modification and scenario-based usability test

The course template site was modified to reflect results from the Cohort B semi-closed card sort.

Five participants from Cohort B who did not participate in the card sort were recruited by email to participate in a scenario-based usability test. Task scenarios were the same as in the first usability test (see Table 1).


Previous | Next