upa - home page JUS - Journal of usability studies
An international peer-reviewed journal

Beyond Specifications: Towards a Practical Methodology for Evaluating Web Accessibility

Panayiotis Koutsabasis, Evangelos Vlachogiannis, and Jenny S. Darzentas

Journal of Usability Studies, Volume 5, Issue 4, August 2010, pp. 157 - 171

Article Contents


Related Work

Research on web accessibility has produced a wide range of results that are used in mainstream web design to promote good design practice. This work can be briefly divided into specifications (and legislation based upon these specifications that aims at encouraging or enforcing the development of accessible web design), approaches, and methods for applying those specifications and tools for automatic evaluation of (aspects of) web accessibility.

Web Accessibility Legislation, Policies, and Standards

Several policies and legislation have been developed either at a national or an international level for promoting and enforcing web accessibility. Countries on both sides of the Atlantic have included eAccessibility into their laws. The U.S. has developed Section 508 of the U.S. Rehabilitation Act (for more information, see http://www.section508.gov). The most recent European policy framework for the information society and media is i2010, which promotes, with the tools available to the Commission, a European Information Society for all citizens (Europe’s Information Society, n.d.). Actions implemented under this priority of i2010 aim to ensure that the benefits of the information society can be enjoyed by everyone (eAccessibility) and encourage provision of better public services (eGovernment and eHealth). Furthermore, there are several national legislations across Europe. Probably the most well known is the Disability Discrimination Act (DDA) 1995 (updated in 2005) in the U.K. and the Act on Equal Opportunities for Disabled Persons (27 April 2002) in Germany (for more information, see http://www.w3.org/WAI/Policy). Laws promoting accessibility are often based on well established de-facto standards, like Web Content Accessibility Guidelines (WCAG) recommended by the Web Accessibility Initiative (WAI) of  the World Wide Web Consortium (W3C).

Currently, most of the countries use WCAG 1.0 as a basis for setting up their local policies. The first version of WCAG guidelines (WCAG 1.0) offers a set of techniques to remedy problem situations. However, these techniques are HTML and web technology centric. We are now at a transition phase as the second major version of WCAG has been released as a recommendation of W3C.WAI (Caldwell, Chisholm, Slatin, & Vanderheiden, 2008). The seven-year (since the first public working draft in 2001) effort to reach consensus on WCAG 2.0 produced guidelines that have tried to address problems present in WCAG 1.0, nevertheless they still present weaknesses. According to several researchers in the field (e.g., Brewer, 2005; Moss, 2006), the guidelines are presented at a very abstract level using general and vague terms; they are difficult to use because they are couched in even more obscure terminology than WCAG 1.0, and they require a great deal of explanation to become comprehensible. Kapsi et al. (2009) found several usability issues of WCAG 2.0 that they suggested could be significantly improved if usability issues could be communicated to the evaluators more clearly. Furthermore, there have been organizational policies like the U.K.’s Royal National Institute of the Blind (RNIB) that highlight some of the important accessibility issues (a subset of WCAG), which are exhaustively presented in Harper and Yesilada (2008).

Apart from the web content, web accessibility depends on two additional components of web development and interaction, including web software (tools) and web developers (people; for more information, see http://www.w3.org/WAI/mobile/experiences). Based on these, the WAI provides complementary guidelines: (a) Authoring Tool Accessibility Guidelines (ATAG and ATAG 2.0 - W3C Working Draft 21 May 2009; for more information, see http://www.w3.org/TR/2009/WD-ATAG20-20090521) and (b) User Agent Accessibility Guidelines (UAAG and UAAG 2.0 - W3C Working Draft 11 March 2009; for more information, see http://www.w3.org/TR/UAAG20). Finally, recently there is much interest on the shared web experience of disabled users, older users (Arch, 2008), and mobile device users.

Web Accessibility Approaches and Methods

W3C describes three evaluation approaches with varying levels of thoroughness and effort:

Several other approaches and methods have been developed to guide the designer or evaluator to apply recommendations.

A systematic methodology for evaluating conformance with the WCAG 1.0 is the Unified Web Evaluation Methodology (UWEM; Nietzio, Strobbe, & Velleman, 2008; Velleman, Strobbe, Koch, Valasco, & Snaprud, 2007). The UWEM is the result of the combined efforts of three European projects that make up the Web Accessibility Benchmarking Cluster (WAB Cluster). Most of the countries that use WCAG 1.0 as a basis for setting up their national policies often have different interpretations of the guidelines. UWEM offers a shared interpretation of WCAG 1.0 in order to provide sufficiently robust tests. In addition UWEM aims at providing a harmonized platform for a European level—common—evaluation methodology that can be used for the development of labeling schemes (for more information, see http://www.euracert.org and ftp://ftp.cenorm.be/PUBLIC/CWAs/e-Europe/WAC/CWA15554-00-2006-Jun.pdf to see the CEN Agreement) and deals with topics such as scoping, sampling, testing, reporting, etc. Some changes are required for the maintenance of UWEM, especially regarding WCAG 2.0.

Another approach, this time aiming to achieve a common methodology on a national level, is that of the U.K.’s PAS 78 BSI Standard (for more information, see http://www.bsi-global.com/en/Shop/Publication-Detail/?pid=000000000030129227). This is intended to be a document that can help the commissioners of web sites to understand accessibility and be able to discuss development with web design project managers with reference to the WAI guidelines, usability testing, automated checking tools, etc. This approach attempts to standardise the process of achieving web accessibility, rather than the properties of web accessibility.

Web Accessibility Evaluation Tools

A large number of software tools have been developed to automatically check web accessibility (for more information, see http://www.w3.org/WAI/ER/tools/). The software tools functionality mainly includes the following:

Because WCAG2.0 was only released a short time ago, there are only a few tools that use the criteria in that version. Presently the ATRC web accessibility checker is the only tool that claims to support it (http://achecker.ca/checker/index.php). However, Imergo® provides added functionality including integration with web 1.0 and 2.0 and advanced crawling capabilities for content management systems and a navigation consistency module (http://imergo.com/; Velasco, Denev, Stegemann, & Mohamad, 2008). Finally, the GOVMov eAccessibility Checker is interesting mainly because it supports UWEM (Velleman, Strobbe, Koch, Velasco, & Snaprud, 2007), discussed in next section.

Despite the plethora of available web accessibility tools, several researchers have identified important limitations including: limited crawling capabilities, the inability to analyze and/or repair dynamically generated Document Object Model (DOM) etc. (Velasco et al., 2008), and inconsistencies between the evaluation results of different tools (Centeno, Kloos, Fisteus, & Alvarez, 2006). This last limitation is basically due to the ambiguity of some checkpoints that in turn lead to different interpretations. This is a problem that WCAG 2.0 has tried to address (see discussion in Kapsi et al., 2009). Also, different assumptions taken from the logic of the tools (like document validation) often lead to very different reports.

In a study aimed to evaluate the effectiveness of web accessibility evaluation tools, Centeno et al. (2006), classified WCAG 1.0 checkpoints into those that were objectively automated, subjectively automated, semi-automated, and manually tested. However, it appears that a reliable coverage comparison of tools would only be possible if the tools were tested on test suites with appropriate metadata (Strobbe, Herramhof, Vlachogiannis, & Velasco, 2006). Therefore, the W3C WAI Test Samples Task Force is working towards creating and harvesting such test suites based on work done by the BenToWeb project (http://www.bentoweb.org). The result reports can then be easily compared by examining the results that are expressed in a specially developed language, the Evaluation and Report Language (EARL) (http://www.w3.org/WAI/intro/earl.php) and its metadata.

In terms of tools to support authoring of content, as opposed to the inspection of content as described above, current authoring tools offer some support to check for web accessibility. The most important effort in this respect is that of Dreamweaver (Adobe) that incorporates a number of features that promote the development of accessible web pages including a built-in accessibility validation tool and accessibility prompts (http://www.adobe.com/accessibility/products/dreamweaver/overview.html). However many web sites are not built from scratch but on top of web application frameworks, for example, content management systems. The major problem with this type of development is that these systems provide little or no support for accessible content generation. The problems with web accessibility design and evaluation tools have become even more intense with the introduction of the so-called web 2.0 technologies as well as with the advent of user generated content; now the roles of the end user and content provider are somehow blended. To try to provide guidance in this area, WAI has introduced WAI Accessible Rich Internet Applications (ARIA) Suite, which uses a semantics-based approach to try to define a way to make web content and web applications more accessible to people with disabilities based on semantics (http://www.w3.org/WAI/intro/aria.php).

Finally, we need to mention that many tools that are not classified under web accessibility evaluation tools may offer useful functionality for either evaluating specific issues of web accessibility (e.g., web developer, http://chrispederick.com/work/web-developer/) or for simulating user situations (like Vischeck, that simulates colour blindness problems http://www.vischeck.com).

Previous | Next