upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

WHAM! POW! Comics as User Assistance

Erika Noll Webb, Gayathri Balasubramanian, Ultan ỎBroin, and Jayson M. Webb

Journal of Usability Studies, Volume 7, Issue 3, May 2012, pp. 105 - 117

Article Contents


Study 2: Conceptual Comics

As a follow up, we wanted to explore the use of comics to explain conceptual information. In Study 2, in addition to a comic, we expanded to include the use of a metaphor in presenting the information. Respondents were asked to view metaphor- and non-metaphor-based comics along with a Microsoft PowerPoint-based form of the same information. In the metaphor-based comic, the metaphor of a sandwich bar was used to illustrate the attributes of the DITA concept. The use of the metaphor drew on a familiar understanding of another concept or idea (Kabapinar, 2005) to assess the hypothesis that such visuals might communicate understanding that the written form cannot ( Andriessen, Kliphuis, McKenzie, & Van Winkelen, 2009). DITA, the comic described, was like a sandwich bar because you could take the same component pieces and create very different end products. McKenzie and Van Winkelen (2011), for example, detailed how visual metaphors facilitated comprehension of knowledge management systems concepts. Use of metaphors and analogies is a complex area, providing a way to integrate human cognitive processes, such as visual perception, logical reasoning, similarity judgment, learning and transfer, creative thinking, and so on (Schwering, Kühnberger, & Kokinov, 2009). Weimann has used the concept of Lego® blocks to explain DITA (2010). As in the previous study, we used the comic templates from DesignComics.org to quickly and easily create the designs for the comics. A few clip art pieces from Microsoft Office completed the designs.

In comparison, we presented the same comic without the metaphor as well as a PowerPoint presentation of the same material. The comic without a metaphor contained the same panels as the first comic, except the reference to the sandwich bar was removed. The PowerPoint contained the same information, including some of the same diagrams that were used in each comic, but no other visuals. These diagrams showed the relationship between DITA elements with boxes, lines, and arrows. In both of the comics, a narrative was used with one character describing DITA to another character. After viewing all types of training materials, participants indicated their preferences for each type on the same two usability scales used in the previous study.

Twenty-seven participants, all technical writers at Oracle Corporation, saw each information presentation in a counterbalanced order: comic with metaphor (Comic/Metaphor), comic without metaphor (Comic/No Metaphor), or PowerPoint. We recruited globally for technical writers who were not currently using DITA for the creation of their material. We had participants in this study from the Americas, Europe, and Asia Pacific regions. Fourteen participants were female, thirteen male. No other demographic data was collected. All of the participants were aware of DITA, and in some cases, had some preliminary training. However, the majority did not know any particular details about how DITA works, or how they might use it.

Figure 3

Figure 3. An example of the comic with the metaphor of a sandwich bar

Each group of nine participants viewed one of the three training materials first, after which they were presented a 12-question multiple choice test to assess their understanding of the information that they saw. All participants viewed all three training materials; however, they were quizzed only on the first material that they viewed. Following the presentation of each of the training materials, the participants were asked to assess their impressions of the materials using the OUS and the SUS.

Quiz Performance

One key question was whether the training materials showed differences in effectiveness. In the first study, there were clear preferences for the comic presentation, but no differences on the ability to perform the tasks in the prototype. In this study, we found some support for the differential effectiveness of the training material.

Table 2. Counts of Quiz Scores With Means and Standard Deviations

Table 2

Participants could have scored from 0 to 12 on the quiz. Table 2 shows that most of the participants scored a 12 or 11 on the quiz, with some scoring 10, and one participant scoring 9. Due to the presence of a ceiling effect in this data, an ANOVA didn’t seem appropriate. One concern was that the standard deviations shrink as the means grow larger, and the largest standard deviation was twice the smallest.

One effect stands out when looking at Table 2. The Comic/No Metaphor condition produced the lowest scores. Four of five (80%) scores below 11 came from the Comic/No Metaphor condition. If we expect 1/3 (because there are 3 conditions) of the scores in the lowest categories to belong to the Comic/No Metaphor condition, then the observed value of 80% is statistically significant (p<.05) using the exact binomial test.

So, the differential effect of training materials on quiz performance can be seen for one condition (Comic/No Metaphor) among the lowest scores. It is interesting to note that Comic/No Metaphor was always second in the ratings data, below Comic/Metaphor and above PowerPoint.

Ratings

Means for the combined OUS ratings were Comic/Metaphor (Mean=6.02), Comic/No Metaphor (Mean=5.70), and PowerPoint (Mean=5.47). The main effect of training material on OUS ratings was not statistically significant at the p<.05 level (F=2.58, p=.10).

Concept-Based Comic Findings

Figure 4

Figure 4. Average OUS ratings for the three training material conditions. Error bars show the 95% confidence interval, corrected for within subjects design using the R statistics package.

There were statistically significant effects of training material on SUS scores (F=3.75, p<.05). The Comic/Metaphor condition showed the highest average SUS score (Mean=79.4), followed by Comic/No Metaphor (Mean=74.4), and PowerPoint (Mean=68.0). The Comic/Metaphor condition produced higher ratings than PowerPoint (t=2.3, p<.05). The advantage of Comic/Metaphor over Comic/No Metaphor (t=1.8, p<.10) and Comic/No Metaphor over PowerPoint (t=1.4, p<.20) did not reach statistical significance at the p<.05 level.

The effect of presentation order on PowerPoint ratings

There appeared to be an effect of the order of presentation of instructional material on OUS and SUS ratings. Combined OUS ratings for the PowerPoint condition got lower as presentation order increased from first to third (Means=6.0, 5.5, 5.0). This order effect (F=2.33, p=.12) was not statistically significant at the p<05 level. A similar pattern was seen in the SUS ratings data (F=2.5, p=.10) as presentation order changed from first to third (Means=77.5, 70.3, 57.5). Though not statistically significant, the trends were nonetheless an interesting observation, suggesting that the preference for PowerPoint decreased as more alternatives were viewed.

 

Previous | Next