upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

A Modified Delphi Approach to a New Card Sorting Methodology

Celeste Lyn Paul

Journal of Usability Studies, Volume 4, Issue 1, November 2008, pp. 7-30

Article Contents


Heuristic review results

Fifteen information architects completed the heuristic review that asked them to rank each of the information structures on a scale from 1 (very poor) to 3 (average) to 5 (very good) (see Table 3). In addition to six heuristics, they were asked to provide an overall rating of the information structure on the 1 to 5 scale, independent of the other information structure (see Table 4).

In each heuristic, the average score of the Modified-Delphi information structure was greater than the Open information structure. The Wilcoxon matched-pair, signed-rank test is used when there are small numbers of matched samples (NIST, 2007) and does not require a normally distributed sample like other tests. The differences are calculated and then ranked by magnitude. (One participant did not fully complete the heuristic review so only 14 pairs were calculated.) The nominal alpha criteria level (α) and the limit for the probability (p) of the outcome under the null hypothesis is set at 0.05. Probability is the measurement of the likelihood of an event.

According to the expert heuristic review responses gathered, the Modified-Delphi information structure is better than the Open information structure in granularity and dimension, consistent and logical naming, and overall, significant at the α = 0.05 level. The comparison of other heuristic responses gathered are not statistically significant based on response distribution and sample size.

Table 3. Average of All Results from Heuristic Review
Heuristic Modified-Delphi Open
Breadth and depth are balanced 4.0 3.4
Labels are clear and meaningful 4.1 3.8
Data is of similar granularity and dimension 4.1 2.9
Naming scheme is consistent and logical 4.0 3.1
Visual hierarchy is clear 4.1 3.6
Organization fits users' needs 4.1 3.4
Overall rating
Individual question, not an average of the heuristic score
3.9 3.2

 

Table 4. Average and Significance of Paired Results from Heuristic Review
Comparison Modified-Delphi Open Probability
(of the difference being significant)
Breadth and depth are balanced 4.0 3.6 0.087*
Labels are clear and meaningful 4.0 3.8 0.461*
Data is of similar granularity and dimension 4.1 2.9 0.004
Naming scheme is consistent and logical 4.0 3.1 0.016
Visual hierarchy is clear 4.1 3.6 0.188*
Organization fits users' needs 4.1 3.4 0.074*
Overall rating
(individual question, not an average of the heuristic score)
3.9 3.2 0.023

* not statistically significant at the α = 0.05 level

Independent information structure ranking

Fourteen participants completed paired responses to the question of usefulness of an information structure. Participants were asked, independent of the other information structure, if the presented information structure would be useful in aiding in the design of an information architecture (see Table 5).

A binomial test is a one-tail test that determines the significance of the deviation from an expected distribution (an equal number of yes and no responses). The tail is the part of the bell-shaped normal distribution curve that is far from the center. Determining that there is no significant difference between the scores is the same as saying that there is a high probability that both scores came from the same distribution. In this case, the scores would fall under the bell part of the distribution and not in the low probability tail.

In a completely random sample, the expected response would be an equal number of yes and no votes for a particular information structure. If the structure is truly helpful in designing an information architecture, the responses would lean towards yes.

The binary test confirms the chance of observing 14 or more yes votes for Modified-Delphi in 14 trials is significant at the α = 0.01 level where p = 0.0001. The chance of observing 12 or more yes votes or 2 or fewer no votes for Open in 14 trials is significant at the α = 0.05 level where p = 0.0065. According to the expert responses gathered, both the Modified-Delphi and Open information structure are helpful in designing an information architecture, significant at the α = 0.05 level.

Table 5. Results from Independent Structure Ranking
Would you find this structure helpful in designing an information architecture? Yes No
Modified-Delphi information structure 14 0
Open information structure 12 2

Previous | Next