upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Dependent information structure ranking

Fourteen participants completed the information structure ranking. Participants were asked to rank which information structure was better to aid in the design of an information architecture (see Table 6). They were presented with two information structures representing results from the Modified-Delphi and Open card sorting studies that were anonymized and order counterbalanced. They were asked to rank the first information structure as being better, the same, or worse than the second information structure. Ten participants responded that the Modified-Delphi information structure was better than the Open information structure. Two participants responded that the Open information structure was better than the Modified-Delphi information structure. Two participants responded that the information structures were the same.

The sign test is a two-tail test based on the assumption that two cases have equal probabilities of occurring. The sign test simply observes whether there is systematic evidence of differences in a consistent direction. This test is ideal for small samples as these differences may prove to be significant even if the magnitudes of the differences are small. The binomial test is a one-tail test based on the probability of observing each case in a series of trials where one answer or the other is selected. All equal observations (where Modified-Delphi information structure = Open information structure) are ignored.

The sign test confirms that the chance of observing 10 or more votes for the Modified-Delphi (or two or fewer votes for Open) in 12 trials is significant at the α = 0.05 level where p = 0.0386. The binomial test confirms the chance of observing 10 or more votes for the Modified-Delphi in 12 trials is significant at the α = 0.05 level where p = 0.0193. According to the information expert ranking, the Modified-Delphi information structure is not equivalent to the Open information structure; the Modified-Delphi information structure is better than the Open information structure, significant at the α = 0.05 level.

Table 6. Results from Dependent Structure Ranking
Ranking Description Number of Votes
Open is better than Modified-Delphi 2
Modified-Delphi and Open are about the same 2
Modified-Delphi is better than Open 10

 

Inverse card sort results

Seven participants completed the Inverse card sort. Participants were asked to select the category where they would be able to find the information asked for in a question (see Table 7).

Both the Modified-Delphi and Open information structures did poorly in the Inverse card sort with very close cumulative scores of 33 and 32 total correct out of 70 possible answers, respectively. Although the Modified-Delphi had more high scores (5 out of 7 or greater) than the Open information structure, it also had the most low scores (1 out of 7 or fewer).

Questions where both the Modified-Delphi and Open information structures did well (5 out of 7 or better) were the following: where to find the final class drop date, the Dean's Notes Newsletter, and school computing services information. Additionally, the Modified-Delphi information structure also had a good score on where to find the Tax Law clinic application, and the Open information structure had a good score on where to find information on the Attorney Practice internship.

Questions where both the Modified-Delphi and Open information structures did very poorly (1 out of 7 or worse) were the following: where to find the Judge Solomon Liss visiting scholars program, a copy of your transcripts, and information about the Student Council. Additionally, the Modified-Delphi information structure also had a very poor score on where to find forms to register for classes at another institution.

There were an insufficient number of responses to conduct a statistical test on the results.

Table 7. Results from Inverse Card Sort
Question # Correct/7 Participants
Modified-Delphi Open
1. Where would you find information about the last day to drop or add a class for the semester? 6 4
2. Where would you read the Dean's Notes Newsletter? 5 5
3. Where would you find the application to the Tax Law Clinic? 5 4
4. Where would you find information about the Judge Solomon Liss Visiting Scholars program? 1 0
5. Where would you request a copy of your transcripts? 1 1
6. Where would you find information about the Student Council? 0 1
7. Where would you find help with connecting your computer to the school network? 7 5
8. Where would you find the form to substitute a class at another institution? 1 4
9. Where would you find information about the Attorney Practice Internship? 4 5
10. Where would you find a copy of the Honor Code? 3 3
Total # correct out of all answered questions 33/70 32/70

Previous | Next