upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Use of Card Sorting for Online Course Site Organization Within an Integrated Science Curriculum

Alison Doubleday

Journal of Usability Studies, Volume 8, Issue 2, February 2013, pp. 41 - 54

Article Contents


In Fall 2011, the University of Illinois at Chicago (UIC), College of Dentistry initiated an integrated, systems-based dental curriculum. The newly implemented curriculum eliminates discipline-based boundaries and focuses on simultaneous investigation of the biochemistry, histology, anatomy, and physiology of organ systems. For example, a student studying the human heart would learn not only the anatomic structure of the heart but also its embryologic development, what it looks like microscopically, how it functions physiologically, how it might be affected by the presence of various microorganisms or pathologies, as well as how this information is relevant, clinically. In an integrated curriculum, all of this content is explored within one course focusing on the cardiovascular system. Students undertake a similar process throughout the first year in classes devoted to the digestive system or to the musculoskeletal system, until all organ systems are explored. Subsequent years revisit this information and introduce additional applications such as pathology, pharmacology, and clinical activities. In contrast to a traditional lecture-based curriculum, small group, case (scenario)-based learning serves as the primary vehicle for instruction in the new model. Collaboration is a key component, and students work in groups of 6 or 7 to research and explore topics introduced through a series of scenarios. In the first year, student work is supplemented with laboratory exercises (such as an anatomy or histology lab) and sessions with content experts. Throughout the curriculum, student schedules are rigid and full with curricular activities running from 8:00 a.m. to 4:30 p.m. daily.

Although considerable planning led to the identification and development of numerous resources available for student use in the integrated curriculum, as the Fall 2011 implementation date neared, the faculty and administration realized that it was essential to develop an intuitive and logical way for students to access these resources. Specifically, the integrated, systems-based approach precludes the division of content into well-defined and isolated disciplines and students must locate and use course materials without the convenience of distinct categories for content organization. Additionally, faculty members from multiple disciplines would need to collaborate to set up and maintain course sites within Blackboard, the course management system used at UIC College of Dentistry. Blackboard is an enterprise software application that provides a means to deliver digital resources to students; students can track their grades, participate in discussion groups and blogs, and take online quizzes, among many other functions. Ultimately, Blackboard course site organization within the new curriculum required a more holistic approach that would parallel the overall spirit and design of an integrated educational experience.

Some challenges for course site organization, imposed by the curricular changes, included the following:

Addressing the challenges listed above and providing appropriate attention to order and consistency within and among course sites were identified as keys to a more effective engagement strategy. As most of the organizational challenges, identified above, related to student usage, and because students spend substantially more time interacting with the course sites than do faculty members or administrators, students were recruited to help determine which specific information architecture conformed best to student organizational preferences and expectations for site navigation. Usability methods, such as card sorting, were employed to obtain these data.

Card Sorting

Card sorting refers to a number of exercises in which participants group and/or name objects or concepts. The data gleaned from card sorting activities allow researchers to understand how participants develop categories or view relationships among concepts (Hudson, 2012). Card sorting grew out of Q methodology, a research method developed in the field of psychology by William Stephenson (1953), a physicist/psychologist. Q methodology arose as a technique for investigating subjectivity and has largely been used in research that hinges upon participants’ perceptions and as an important methodology for qualitative analysis (Brown 1996). Q sorting refers to the ranking or grouping of variables by participants and is the means by which data is generated for factor analysis in Q methodology.

Within the field of psychology, card sorting has been used for making inferences about participant characteristics, such as reaction time (Jastrow, 1898), memory function (Bergstrom, 1893; Bunch & Rogers, 1936), and personality assessment (Block, 1961). Later applications of sorting activities include the development of the Wisconsin Card Sorting Test (Berg, 1948), a neuropsychological test, often used clinically for identifying frontal lobe impairment (Anderson, Damasio, Jones, & Tranel, 1991) and, more recently, the use of card sorting for structuring cartographic symbols (Roth et al., 2011,). In recent years, card-sorting methodologies have frequently been employed in usability research (Nielsen & Sano, 1995) and in web design (Lewis & Hepburn, 2010). Card sorting provides a level of insight and understanding into the mental models of website users by examining how users categorize and organize website material. These data allow site developers to display content in such a way that the extraneous cognitive load required to navigate the site is minimized, and users can retrieve information from the site more easily.

Types of Card Sorting Methods

There are many different types of card sorting methods that may be employed during the design process. Which methodology is selected depends very much on the questions being addressed and, often, on the stage of design being investigated.

In an open card sort, a participant groups together cards that seem to logically belong together, and it is the participant who, ultimately, defines and names all grouping categories. Open card sorts provide a great deal of freedom to participants, allowing them to eliminate cards, to sort cards into as many groups as desired, and to develop names for each grouping.

In a closed card sort, the user is asked to group cards into predefined categories. This technique provides more structure and less flexibility to participants. This type of approach can be very useful if the goal is to modify an existing structure.

A semi-closed card sort is a variation on the closed card sort. It is conducted, initially, as a closed card sort using predefined categories but the participants can change the category names at any time.

An inverse card sort is another variation on the closed card sort and asks participants to find or rank cards within a completed structure. This strategy is often used to validate or test assumptions about information architecture.

Additionally, there are other variations, such as a Delphi method or a modified Delphi method (Paul, 2008) that allow participants to receive some feedback about the card sorting results of other participants prior to doing their own sorting.


Previous | Next