You are here: Home > Publications > Accessibility evaluation practices

Accessibility evaluation practices - survey results

Summary

This is a summary of the results of an online survey on accessibility evaluation practices conducted in late 2004. Web accessibility practitioners were invited to participate in the survey via a posting to several key mailing lists. There were 98 respondents, with the majority from the IT&T and education sectors whose roles involve less than 50% accessibility-related work. Manual inspection was the most popular evaluation method used, and user testing the least popular. Most evaluations were based on the World Wide Web Consortium's Web Content Accessibility Guidelines (WCAG).

Introduction

In preparing a conference keynote presentation on the relationship between usability and accessibility, I was interested in getting a better understanding of how accessibility evaluations are conducted. I had some anecdotal evidence based on the work of my own team and others with whom we collaborate, but I wanted a better picture of how often user testing is involved in accessibility evaluation, what sorts of users are recruited to participate, and how testing is approached.

I prepared a short online survey and posted some messages on several mailing lists (WAI interest group list, WebAIM discussion forum, Web Standards Group discussion list, and the CHI-web mailing list) inviting people involved in web accessibility to respond. The survey ran from October 31 to November 11, 2004.

Respondents

There were 98 respondents to the survey. Not all respondents answered every question, and some responses were unclear.

Industry sectors represented in the data

The survey respondents were primarily from the IT and telecommunications and education sectors, with government and the finance/insurance industry also represented.

IT and telecommunications (44%)
Education (33%)
Government (11%)
Finance/insurance (10%)
Other (3%)

Note: four respondents gave unclear answers, and the percentages above do not include these respondents.

Percentage of role devoted to web accessibility

Less than a quarter of the survey respondents spend more than half their time on web accessibility-related activities.

Less than 25% of role (42%)
25-50% of role (35%)
51-75% of role (10%)
More than 75% of role (12%)

Standards for evaluation

Almost three quarters of the respondents base their accessibility evaluations on the W3C's Web Content Accessibility Guidelines (WCAG).

WCAG (56%)
Section 508 (26%)
Both WCAG and Section 508 (16%)
Other (2%)

Accessibility evaluation methods

Tools to automate evaluations

A quarter of the respondents indicated that they never, or only rarely use tools to automate accessibility evaluations, but almost half use them routinely.

Rarely/never (26%)
Sometimes (29%)
Usually (26%)
Always (19%)

Manual inspection methods

Manual inspection methods are widely used. Only 10% of respondents said they rarely, never or only sometimes use manual inspection methods during accessibility evaluations.

Rarely/never (2%)
Sometimes (8%)
Usually (29%)
Always (61%)

User testing

User testing is not commonly used in accessibility evaluation. Almost three quarters of the respondents said they never, rarely or only sometimes involved end users in testing accessibility.

Rarely/never (41%)
Sometimes (32%)
Usually (20%)
Always (7%)

How user testing is carried out

Disabilities represented in user testing populations

All respondents indicated they usually included users with visual disabilities (either blindness or some other visual impairment) in user testing. Just over half said they usually included users with mobility-related impairments. Only 13% said they usually included users from all disability groups.

Blindness (87%)
Low vision and other visual impairments not including blindness (80%)
Hearing impairment (25%)
Motor/mobility impairments (54%)
Cognitive impairments (26%)
Seizure disorders (11%)

Note: The percentages above are based only on those who answered this question (61 respondents).

Technological competence of users

When asked if they usually used only competent users of adaptive/assistive technologies in user testing, almost two-thirds of respondents indicated they did.

Yes (65%)
No (35%)

Sampling and familiarity of users

When asked if they usually used the same users for most of their user testing, just over half of the respondents indicated they did.

Yes (57%)
No (43%)

Note: due to a scripting error, the first 39 respondents answers to this question were lost. The percentages above represent the answers of only 37 respondents.

Approach to testing

Task-based testing was the most popular method used, with almost two-thirds of respondents taking this approach.

Page checks (63%)
Tasks (30%)
Other (7%)

Respondents' comments

There were a range of comments, but three main issues came up: it is difficult to recruit users for testing, resourcing is problematic, and sometimes a lack of management support prevents practitioners from being able to make sites and applications more accessible.

Recruitment issues are a barrier to user testing

"User testing is very tough because the population of available (users) is very small."

"Finding a representative sample of people with selective dsabilities who also have the correct skill set to use the software (trained in business functions) is next to impossible."

"We try to user test, but we have only situational observations of a small user group."

"Difficult recruiting for any other users than the visually impaired."

"I believe user testing is a good approach. The only problem is in finding enough volunteers."

"I would prefer to use a broader range of users, but they are hard to find."

Resourcing is problematic

"... I do not have the resources to engage users (with or without disabilities) in any thorough user testing."

"Would like to have a mobility-impaired group especially (for user testing), however I have no budget to compensate testers."

"With a site which is as large as ours we are finding it difficult to implement these very necessary changes site wide."

Lack of management support

"We would like to employ user testing in our evaluations, but we are having problems justifying the expense of extra testers to the powers that be."

" Prior to this I have been working in the private sector where management was not interested in 508."

"Since I work on applications to support internal areas of my company, they do not place a significant focus on developing applications that are accessible to all audiences."

Conclusions

Manual inspection for conformance with the W3C's Web Content Accessibility Guidelines (WCAG) is the most common approach to accessibility evaluation, based on the data collected in this survey. User testing is the least-used of the three evaluation methods.

For those who do employ user testing, the focus is on people with visual disabilities. This probably explains the strong trend towards recruitment of those who are competent users of assistive or adaptive technologies such as screen readers and may also a factor in the use of the same users in various testing sessions.

Many comments related to user testing. Several said that recruiting participants for testing was difficult, but others faced problems with management support and resourcing.