Cabinet Office
The methodology comprised the following steps in order to check conformance with the WCAG 1.0 recommendations.
All of the European website addresses supplied were subject to automated assessment. The automated assessments were carried out by Greytower Technologies using a suite of software tools developed by them and used in previous projects with RNIB, Socitm and AbilityNet.
A selection of the sites that underwent automated assessment was then subject to manual assessment and manual verification of the automated data. A team of web accessibility experts from RNIB, AbilityNet and Dublin City University carried out the manual assessments and data verification.
Those sites that achieved Limited Pass Level A were used as the start of a final step in selecting sites that might be considered as representing good practice.
The initial list of URLs was taken from three sources:
The URLs from all three sources were checked for validity, and consolidated into a single list of 569 URLs, which were submitted for automated assessment.
Greytower Technologies, a web consultancy based in Sweden and the UK, which specialises in web accessibility, and which RNIB have worked with successfully for several years carried out the automated assessment. The company has developed an in-house suite of software tools for checking websites for a wide range of issues, with a focus on accessibility. The project team drew up a list of assessment and data output requirements, and all sites assessed using that specification. This assessment specification was based on and built on the Level A specification developed for work on the Socitm Insight Better connected annual reports on UK local authority websites. The specification includes analysis of 60 distinct characteristics of each retrieved page. Overall, these address seven distinct WCAG 1.0 Priority 1 checkpoints and ten distinct Priority 2 checkpoints.
Each of the submitted 569 URLs was processed by first automatically retrieving a number of pages, starting from this home URL and progressively following hyperlinks to a pre-programmed depth (within each domain/site). The retrieved pages were then each subject to the automated assessment as specified. The results were consolidated for each site and stored into a spreadsheet for detailed analysis. At this stage a number of sites were removed from further consideration, where sufficient data was not obtained for reliable investigation. This could arise for a variety of reasons:
At the conclusion of this process, usable data was obtained for a total of 436 sites. These were consolidated, in turn, into a smaller set of fully automated conformance checks (five relating to two distinct Priority 1 checkpoints and twelve relating to nine distinct Priority 2 checkpoints), together with a range of conformance indicators that can inform and guide more detailed manual checks of other checkpoints, which cannot be fully automated.
The overall total number of manual assessments possible was set by the resources available within the project. A selection process was designed to ensure effective validation of all the automated check results (both positive and negative), and also to ensure a good pool of candidate sites for identification of good practice. The process adopted is described below.
For each EU country, including some European institutions as a nominal ‘Member State’), the best and worst sites were identified, on the basis of the automated data. The best site was identified as being the site that had actively passed the most data checks. The worst site was identified as being the site that had actually failed the most data checks.
For countries with more than 20 valid automated results in the list of sites, both the best and worst sites were passed forward for manual inspection. For countries with fewer than 20 valid automated results in the list of sites, just the best site was passed forward for manual inspection. This resulted in 31 sites being selected
As part of the manual assessment, the details of the data reported by the automated assessment were checked to ensure the accuracy of the results of the automated assessment. The detailed specification for verification of automated data can be found at www.cabinetoffice.gov.uk/e-government/eaccessibility/ as part of the supporting documentation for this project.
Building on the results and data provided by the automated assessments, the team from RNIB, AbilityNet and Dublin City University carried out additional manual assessments. Detailed guidance notes, and an assessment spreadsheet with standardised selection lists for responses, were drawn up and circulated to all involved in the manual assessments. Before the start of the manual assessments, one site was selected for a trial assessment in order to ensure coverage of as many issues as possible. All assessors carried out a manual assessment of the trial site. The results were compared, and those issues that resulted in markedly differing assessments were discussed, and agreement reached on how these issues should be assessed and judged.
A second trial site was selected and assessed by all assessors, and again the results were compared. This time the results were sufficiently consistent, and the manual assessments went ahead.
The assessment spreadsheet for each site contained the data from the automated assessment for that site, the example URLs furnished by the automated assessment for that site, and a results spreadsheet for the assessor to enter the results of the manual assessment. Room was provided for comments relating to specific URLs, and for comments relating to the checks being carried out. The detailed specification for manual assessment carried out on each website can be found at www.cabinetoffice.gov.uk/e-government/eaccessibility/ as part of the supporting documentation for this project.
When all the manual assessment results were returned, they underwent extensive additional manual checking by a single senior assessor to ensure consistency and accuracy.
The list of sites that reached Limited Pass Level A were ranked according to the number of passes and fails. A mini-audit was performed on each of these sites to elicit areas of good practice using the following criteria: