Cabinet Office
This project conducted automatic testing of the accessibility of 436 government websites across the EU supported by manual testing across a sample of these. Consistent with similar studies that have taken place, the results found relatively few sites that achieve even Limited Pass Level A conformance with the W3C Web Content Accessibility Guidelines (WCAG 1.0). The project researched correlations between eAccessibility and policy, using information on the many potential factors for influencing success captured in the policy survey. The potential link between eAccessibility and online sophistication was also explored.
The main objective of this study is to report conformance of government websites in the EU with the W3C WAI Guidelines (WCAG 1.0). The study successfully evaluated 436 websites owned by public administrations of the EU’s 25 Member States and several sites owned by the EU. The overall conformance with Level A is shown in Chart 3 below.
This data is based on a variety of accessibility checks derived from the WCAG 1.0 Priority 1 checkpoints, i.e. relating to Level A conformance. It was possible to fully automate a number of these checks, and these were comprehensively applied to an extended selection of pages from every site. (This extended selection of pages was generated by a computer simulating the browsing behaviour of a person, following hyperlinks from the home page to a pre-programmed depth.) The remaining checks required human judgement and interpretation and were carried out on a smaller sample of sites (31 in total). Results from these manual checks were then extrapolated to the full set of sites where appropriate.
A summary of the W3C WAI Guidelines including the four-stage system for classifying passes and failures is given in Appendix 1, followed by a detailed description in Appendix 2 of the evaluation methodology used in this study. Appendix 3 discusses the role of automated evaluation, which is part of this methodology.
The chart shows that, based on this methodology, approximately 3% of the sites achieve Level A conformance, passing the full suite of both automated and manual checks. A further 10% of sites fully passed all the automated checks, but showed a material failure on one or more of the manual checks (Limited Pass Level A). Another 17% of sites failed one or more of the automated checks, but this failure was limited in extent or scope (Marginal Fail Level A). Finally, the remaining 70% of sites showed widespread failure against one or more of the automated checks (Fail Level A).
A number of automated and manual checks derived from WCAG 1.0 Priority 2 checkpoints, i.e. relating to Level Double-A conformance, were also carried out. While many sites passed at least some of these further assessments, no sites were identified which yet achieve the comprehensive level of accessibility that would be indicated by full Level Double-A conformance.
Chart 3 Conformance with Level A (WCAG 1.0)
There is, therefore, a continuing high failure rate as measured by WCAG 1.0 conformance, but for those who have studied recent research these results will be of no surprise.
For example, a comprehensive assessment using a very similar methodology of websites across four EU states, and reported in July 2004, found that only approximately 5% (out of 4349 sites) achieved Limited Pass Level A conformance. It is important to note that this assessment included a wide range of both public and private sector sites. (Footnote 1).
In contrast, a comprehensive assessment using an almost identical methodology into local authority websites in the UK and completed six months earlier in December 2004 found that a much higher 13% (out of 468) achieved Level A conformance (Footnote 2).
Nonetheless, it should be emphasised that 3% of sites in the current study did already achieve essentially full conformance with WCAG 1.0 Level A in this study. This demonstrates that it is a very real, immediate and practical objective to achieve. Moreover, if we consider the 10% of sites which already pass all the automated assessments, and the additional 17% of sites, which had only limited failures on these automated assessments, there are a large number of sites, which have already made significant progress towards this level of conformance, and which may require only relatively modest enhancement in order to achieve it fully.
This survey focuses on the public sector. The results show that much has to be done to make services e-accessible, but how typical are they across other sectors? One of the partners in this survey (AbilityNet) has in the past two years applied similar methodology, but in less depth, to assessing 76 websites in commercial sectors in the UK (airlines, newspapers, banks, supermarkets, sport, retail and telecommunications) as part of the quarterly ENation review. Of these 76 sites only five have achieved Level A. This equates to 7% of the sites tested, compared with 13% in this survey that achieved Limited Level A., or better.
This conclusion, therefore, suggests that the private sector is certainly no further advanced than the public sector when it comes to web accessibility.
Set a clear target for making all public sector websites in the EU conform with WCAG 1.0 Level Double-A by 2010 as part of the i2010 strategy to promote an inclusive European information society.
See Section 5.2 for full list of recommendations
A Comparative Assessment of Web Accessibility and Technical Standards Conformance in Four EU States. Carmen Marincu and Barry McMullin. First Monday, volume 9, number 7 (July 2004). http://eaccess.rince.ie/white-papers/2004/warp-2004-00/warp-2004-00.html [External Website]
Better connected 2005: a snapshot of all local authority websites (Socitm Insight www.socitm.gov.uk/insight) [External Website]
The distribution of failure rates is also of interest as Chart 4 below shows: Note: The total comprises 25 Member States plus the European institutions.
Chart 4 Conformance with Level A (WCAG 1.0) by Member State
The results vary from one Member State to another. The state with the highest percentage of Limited Passes at Level A achieves 40%, but nine states have no passes at all. If we extend the classification to include those we have classified as Marginal Fail Level A, then four achieve 50% or more (the highest 60%) but five Member States still have fewer than 10%.
The eight Member States that achieve the highest proportions of sites classified as Marginal Fail Level A and higher are shown in Chart 5.
Chart 5 Member States with highest levels of conformance
Why do some Member States do so much better than others? Some clues are given in Section 3.3, which examines the correlations with potential influences.
The policy survey requested respondents to estimate the proportions of sites in each Member State that already conform with WCAG 1.0 at various levels. However, only six respondents considered that they had adequate information to make such estimates. The most comparable data returned related to Level A conformance within the public sector, where there were six clear estimates. These estimates and the actual results at Limited Pass Level A are shown in Chart 6.
The key lessons to be learned from this chart are, firstly, that most Member States are not even in a position to estimate their conformance and, secondly, that those who did provide them have offered low estimates, which clearly indicate that they know they are not coming up to even the Level A standard. Moreover, in the six cases where we can compare actual figures with estimates, five significantly over-estimated performance and three have no positive results to report at all, according to our testing of the selected sites.
If we assume that the sites studied are representative of government websites in each Member State, these observations all show that there is a significant gap in information and feedback between policy planning and actual outcomes. Closing this information gap should improve effectiveness of all related policy interventions. This could be realised independently by Member States, but there may also be benefits and synergies through co-operative efforts at the European level. The European Internet Accessibility Observatory Project (EIAO) (currently supported under the EU Sixth Framework Programme) may provide a mechanism for such collaboration. Its stated goal is to ‘contribute to better eAccessibility for all citizens and to increase the use of standards for online resources’ (Source: www.eiao.net ). [External Website]
Develop feedback mechanisms for closing the information gap between policy planning and actual outcomes across the EU (The European Internet Accessibility Observatory Project (EIAO) may provide a mechanism for such collaboration).
In particular, aim to test systematically, and on a regular basis, the progress that should now be made. This may include the revisiting the current study in 12 to18 months’ time, and should be integrated with ongoing i2010 monitoring activities.
Develop feedback mechanisms for closing the information gap between policy planning and actual outcomes so that regular monitoring of performance against eAccessibility is made and communicated.
See Section 5.2 for full list of recommendations
The policy survey identified ten potential factors that might influence web accessibility. The correlation of each of these (plus the aggregate index of engagement) with the proportion of sites achieving Marginal Fail Level A or better was calculated. Chart 7 identifies the two individual influences that were then judged to show a statistically significant correlation (98%+ confidence level).
Chart 7 Accessibility results correlated with influences
From this chart it is seen that one single influence, which was found to have the strongest correlation with accessibility outcomes, was the existence of incentives (legal or otherwise). This was found to be statistically significant, but it should be acknowledged that this correlation is still relatively weak in absolute terms (at 0.46 on a correlation scale of zero to one) and should not be over-interpreted.
The detailed answers about the nature of the incentive are reproduced in Section 2 of this report. Eight of the nine Member States where some form of incentive exist also have a legislative framework that reinforces the incentives available, the exception being Denmark where an annual competition drives up standards. Overall, the result does suggest that Member States could usefully collaborate on sharing practice in the types of incentives that they have in place to encourage stronger commitment to accessible web design.
Review the incentives available to encourage the provision of accessible websites in the public service and, if necessary, consider the need for a strong legislative framework.
See Section 5.2 for full list of recommendations
The influence showing the second strongest individual correlation with accessibility was the reporting of recent local studies on web accessibility in the relevant jurisdictions by fifteen Member States. While this is again only a modest correlation in absolute terms, it does suggest that there is a degree of positive interaction between local studies and actual accessibility outcomes. Further, it seems likely that this wealth of evidence from such local studies contributes to the enhancement of web accessibility practices across Europe e.g. via the European Design for All e-Accessibility Network (EDeAN).
The third bar of the chart shows the impact of positive answers to all ten factors, which together can be seen as an index of engagement, with the web accessibility agenda. This shows a strong, overall correlation with current outcomes.
Chart 8 A comparison of policy engagement with eAccessibility outcomes
Chart 8 shows in greater detail the relationship between the percentage of sites in each Member State achieving at least Marginal Fail Level A conformance and this engagement index. The total set of influences surveyed give a meaningful measure of an index of engagement of each state with web accessibility policies. Stronger overall policies are already showing stronger outcomes in terms of actual WCAG 1.0 conformance. However, the particular mix of influences can vary significantly. Different Member States may choose to focus on different influences, but still achieve comparable outcomes.
These findings are also illustrated in Chart 5 (see section 3.1) that lists the eight Member States where government websites have at least been classified as Marginal Fail Level A or better in 40% of cases examined. The policy survey shows that five of these eight Member States have a higher than average number of examples of engagement. In fact, the top three have the three highest number of mentions. Moreover, seven of the eight have some form of incentive in place (out of nine Member States in total that have some form of incentive), suggesting that the existence of incentives might lead to more accessible websites.
Produce a plan for improving awareness throughout the country by reviewing all the examples of engagement identified in this survey as ideas for improving awareness of web accessibility.
See Section 5.2 for full list of recommendations.
The majority of the sites studied are those put in place to provide the e-government services which are benchmarked each year as part of the eEurope initiative. These are, generally, government websites which offer some degree of interaction or transaction rather than purely informational sites. Given this situation and the general move towards greater degrees of transaction in e-government services, we explored the possibility of a link between e-government sophistication and web accessibility. More sophisticated websites might be harder to make and keep accessible, or conversely encourage much greater awareness.
In making this correlation, the study team used the eEurope/Cap Gemini definition of eEurope sophistication (see report entitled Online Availability of Public Services: How is Europe Progressing?) [External Website].
In the event we identified no statistical correlation between accessibility and either of the key measures used in that report for online sophistication or the availability of government information online. It seems likely that the main reason for this lies in differences between the methodologies of site sampling used in the two studies. The link between website sophistication and accessibility cannot be ruled out. On the contrary, both principled technical considerations and qualitative experience (including some of the manual evaluation in the current study) suggests that more sophisticated sites do pose a wider variety of potential accessibility barriers than less sophisticated sites.
However, the guidelines do not by definition indicate that higher complexity levels automatically imply worse accessibility for a site. It could be argued that the underlying content management system is, in that respect, far more important. A well-designed content management system can enforce adherence to a significant number of checkpoints and assist in meeting a range of others.