Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility
Panayiotis Koutsabasis, Evangelos Vlachogiannis, and Jenny S. Darzentas
Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 157 - 171
Article Contents
Method
The method used to evaluate the accessibility of e-publishing web sites includes the following two basic steps that are a part of any comprehensive web accessibility evaluation process.
- Use of automated accessibility evaluation tools that check the conformance of a web page according to the WCAG 1.0 (http://www.w3.org/WAI/intro/wcag)
- Expert (manual) inspection of accessibility on the basis of heuristics that complement automatic evaluation
For the purposes of the accessibility evaluation of scientific e-commerce web sites, the following decisions were made:
- The homepages of e-publishing web sites were evaluated in order to give an idea of the current status of accessibility. The homepage is the first page users will encounter when entering the site.
- Two widely accepted tools were selected to perform the automated tool-based aspect of the evaluation: HiSoftware® Cynthia Says™ (http://www.cynthiasays.com/) and Fraunhofer FIT’s Imergo® (http://imergo.com). The evaluation followed WCAG1.0 because there is currently no tool that can support the automatic evaluation of web accessibility according to version 2.0 of the guidelines.
- A set of heuristics to support human judgment in order to perform expert (manual) inspection of web accessibility were selected. These heuristics were proposed by W3C.WAI and are commonly used in accessibility evaluations and refer to viewing the web site under simulated constrained conditions, including
- disabled colours,
- disabled style sheets, and
- disabled images.
A practical way to perform these heuristics is to use the Mozilla Firefox browser with the web developer toolbar, which enables turning off/on various web design elements to ensure well-formed design and accessibility.
The aforementioned approach to accessibility evaluation is sufficient in order to give an overview of accessibility problems and indicate the major areas that need improvement. This approach integrates basic steps that can be applied in a time-saving way by evaluators who do not need to be proficient with web development technology. However, this approach is not a complete account of the steps required for a complete web accessibility evaluation process, which is presented later in this paper.
Summary of Accessibility Problems Found
Table 1 shows the accessibility problems of scientific e-publishing homepages identified by the automated evaluation tools used. The results of automated evaluation can vary between web accessibility tools due to their differences in the implementation of automated checks. In particular, Imergo seems to make a more detailed automated web accessibility check in comparison to Cynthia Says with respect to some aspects of WCAG double-A and triple-A priorities.
Table 1. Accessibility problems of scientific e-publishing homepages identified by automated evaluation tools (Cynthia Says, HiSoftware & Imergo, Fraunhofer, FIT).
The overall level of web accessibility for the scientific e-publishing homepages is not satisfactory. All 10 homepages examined had some accessibility problems, however these vary. Some present few accessibility problems, e.g., H2 (Homepage 2) presents five accessibility problems in total when evaluated by the Cynthia Says tool and four problems when evaluated by Imergo. On the other hand, the automated web evaluation for some of scientific publishing homepages is quite disappointing. For example, H1 presents a total of 115 problems when evaluated by Cynthia Says and 246 when evaluated by Imergo. The homepages that present many web accessibility problems are totally inaccessible and would need considerable redesign effort to reach a good level of accessibility. Overall, three sites have quite a large number of problems (ranging from 24 to 106!), five homepages have a few accessibility problems, while two homepages are clean at the single-A level. Similarly, the double-A and triple-A levels of accessibility evaluation also reveal a considerable number of problems for most web sites.
As part of the second step of the method, Table 2 provides an overview of accessibility problems identified by human judgment based on the heuristics discussed above.
Table 2. Accessibility problems of scientific e-publishing homepages identified by human judgment found after applying simple heuristics.
The accessibility problems that were found in scientific publishing homepages include the following:
- Failure to provide alternative text for graphical information: In the cases where images are solely used to convey important information for comprehension and navigation (such as links to other pages, texts, and diagrams), the information conveyed by images will not be available to constrained access contexts, (e.g., users with visual impairments, users with low bandwidth, and users with minimal content display devices).
- Failure to provide alternate and/or redundant text and required associations for supporting user input: Forms are not accessible unless they include alternative text for describing the information requested from the user and associate labels with form controls.
- Failure to allow the user to control the interaction of interactive elements of the page: One of the most traditional usability principles is that the user should be in control of the interaction at all times; this is not true with some inaccessible interactive web technologies such as animated images and auto-refreshing of the contents of a web page.
- Failure to provide consistent and effective navigation when accessed under a simulated constrained context: Many homepages suffer from significant navigation problems such as the disappearance of navigation bars (e.g., when they are designed as images with no alternative text), same colour of text with background (when colours are turned off), and others.
- Failure to provide consistent and meaningful information organisation when accessed under simulated constrained context: Some homepages suffer from significant disruption to the organisation of information, especially when style sheets are turned off.
Again, it is evident that there are significant accessibility problems in the scientific e-publishing homepages and that not one homepage was judged as adequately designed to meet all the criteria offered by the heuristics. Six out of ten e-publishing web homepages have at least one major accessibility problem. Also, all sites have minor accessibility problems. These problems are different for each homepage and mainly concern information organisation, navigation, visibility, and user control.
A comprehensive accessibility evaluation cannot stay at the level of homepages but must also include other sections of the scientific e-publishing web sites. However, given that the homepages of scientific e-publishing web sites are not accessible, it is quite safe to assume that the rest of the application will include at least the same types of problems. This example demonstrates that even the contemporary scientific publishing sites present a wide range of accessibility problems. Scientific publishing homepages will reveal several problems when accessed in constrained situations, such as, access from mobile devices, black and white print-outs, access from voice browsers, access from assistive technologies used by people with motor disabilities, and so on.