upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

WHAM! POW! Comics as User Assistance

Erika Noll Webb, Gayathri Balasubramanian, Ultan ỎBroin, and Jayson M. Webb

Journal of Usability Studies, Volume 7, Issue 3, May 2012, pp. 105 - 117

Article Contents


Study 1: Task-Based Comics

Nine managers who had conducted performance evaluations of their direct reports participated in the study conducted at the Alliance 2011 conference (an Oracle User Group Conference). These managers were technical group leaders at public universities who were responsible for the integration of Oracle products for use in their organizations. They were not usability experts. There were four women and five men. Other demographic information was not collected.

The participants were shown a PowerPoint presentation in which there were slides describing a new feature for performance evaluations in Human Capital Management (HCM) software. The participants were also shown a comic with the same information about the new feature (see Figure 1), with the exception that the comic added a narrative between two characters discussing the feature. One character described and demonstrated the new feature to the other character. The comics were developed using a set of characters developed by the I See Design (ISD) Group that were made available on Design Comics (http://www.designcomics.org/).

The order of presentation was partially counterbalanced. Following each presentation, participants used the HCM software prototype to perform tasks described in the materials that they viewed. Finally, participants were asked to evaluate the materials on two usability scales. The first was a four-question software usability scale used internally at Oracle that was modified for the presentation of training material, substituting the words “training materials” for the word “software.” This scale asked about the perceived ease of use, attractiveness, usefulness, and understandability of the materials. The second, also modified for training material instead of software, was the System Usability Scale (SUS; Brooke, 1996). The SUS is a simple, widely used 10-statement survey. It was developed by Brooke in the 1980s as a means of getting a quick subjective measure of system usability. Users are asked to rate their level of agreement or disagreement to 10 statements about the software. In our case, we replaced “software” with “training materials,” but otherwise left the scale the same. Because 5 of the 10 questions were worded negatively and the other 5 were worded positively, we used an Excel scoring template that turns the raw individual survey ratings into a single SUS score. This template was based on Brooke’s standard scoring method (ratings were standardized to a common 0-4 rating, then multiplying the sum by 2.5 to get a score that can range from 0-100). The participants were able to perform all tasks in the prototype (that is, success rates were 100%). There were no other differences in performance or perceived usability in this study. The focus for this study was on the users’ perceptions of the usefulness and usability of the different types of training materials.

Figure 1

Figure 1. Examples of the PowerPoint presentation (top) and a page from the comics (bottom) showing how to perform a task in the HCM software

Task-Based Comic Findings

Overall, participants preferred the use of the comics to present task-based information. We measured this preference first using the Oracle Universal Scale (OUS). This scale has four questions that are answered by choosing a number from 1 (less of the quality) to 7 (more of the quality):

Figure 2

Figure 2. Average OUS question ratings for Comic and PowerPoint conditions. Error bars show the 95% confidence interval, corrected for within subjects design using the R statistics package1. Error bars are truncated at the maximum rating of 7. An * next to the OUS Question name indicates that the difference between means was statistically significant at the p<.05 level.

The four OUS questions were averaged together to form a single usability scale. Comics were rated higher than the PowerPoint presentation (see Table 1, Overall Average) and this difference was statistically significant. Looking at the individual OUS questions, the advantage for comics was statistically significant for ease of use and understandability ratings, but not for attractiveness and usefulness ratings.

Table 1. T-Test Results (two-tailed) for the Effect of Information Format (Comic vs. PowerPoint) on OUS Ratings

Table 1

The SUS scores showed a large and statistically significant preference for Comics (Mean=88.1) over PowerPoint (Mean=43.6; t=8.63, p<.01).

So, both the OUS and SUS scales showed that users preferred Comics over PowerPoint.

Affective reactions

Participants in this study had strong reactions on an affective level to the two types of materials. During and after the sessions, participants discussed their perceptions of the comics in very different terms than they used for the PowerPoint presentations. The contrast was especially striking because we did not specifically probe on these issues unless the participant brought it up. Participants used terms like “friendly,” “easy,” “accessible,” “engaging,” and “fun.” In contrast, in discussing the PowerPoint, participants used more negative language, including that they felt the material was “dry,” “formal,” “overwhelming,” “threatening,” “frightening,” and “intimidating.” Our interpretation was that this affective response shows that the comics appealed to users at an emotional level. One participant said of the PowerPoint, “I don’t know if I would use the new feature because it was intimidating… Before I went into the app I was already stressed.” While in response to the comic, one user expressed the advantage of the comics particularly well, “A comic doesn't make you afraid of it... how hard can it be if there's a little comic strip?”

1http://tinyurl.com/82phmk8

 

Previous | Next