This is the third part of the “Accessibility of municipal websites in Norway after Web Accessibility Directive” series posts. In first part, “Accessibility of municipal websites in Norway after Web Accessibility Directive – introduction” , I reflected on motivation, methodology and such and in the second part, “Accessibility of municipal websites in Norway after Web Accessibility Directive – statements analysis“, I summarized the accessibility statements analysis.
In this, third, part I will summarize quick automatic tests made on homepages of all 356 Norwegian municipality webpages.
Automatic accessibility testing of homepages
As I mentioned in the first part I used some free time to do all of this and am now only documenting it in these posts. Proper automatic testing of all websites would take some time and effort, but a quick check for all 356 homepages was totally doable. When testing more pages we would for sure detect more probable accessibility issues, that’s a fact. Some would be duplicates, some would be a minor variants of same nature (so almost duplicates) and some would also be false positives (yes, even automatic accessibility testing can sometimes be wrong).
WCAG Success Criterion | # |
---|---|
4.1.2 Name, Role, Value (A) | 223 |
2.4.4 Link Purpose (In Context) (A) | 95 |
1.4.3 Contrast (Minimum) (AA) | 89 |
4.1.1 Parsing (A) | 54 |
1.1.1 Non-text Content (A) | 44 |
1.3.1 Info and Relationships (A) | 19 |
1.4.1 Use of Color (A) | 11 |
3.1.1 Language of Page (A) | 5 |
1.4.4 Resize text (AA) | 2 |
2.4.2 Page Titled (A) | 3 |
3.1.2 Language of Parts (AA) | 1 |
The tool detected 546 WCAG A and AA issues on all 356 homepages.
I’ve left out WCAG AAA issues detected and I’ve also left out best practice “failures” detected by automatic accessibility testing tool as they aren’t relevant at the moment.
We can quickly detect the limitation of automatic accessibility testing tools as they can only find some of WCAG failures – there are way less different WCAG success criteria found by automatic tools. But on the other hand – we can also see that some of automatically detected WCAG failures are quite popular considering small number of sites tested. It is the obvious code-based ones that prevail when doing accessibility testing. And that’s quite logical as automatic accessibility testing is code testing code.
Only 5 out of 356 homepages didn’t have any automatically detected WCAG failures. This is strange when we remember the numbers from previous post, “Accessibility of municipal websites in Norway after Web Accessibility Directive – statements analysis“, where statement analysis claims that 11 municipalities conform to WCAG 2.1 on A and AA.
Only 2 out of 11 that claim full conformance didn’t have automatically detectable failures
As mentioned in the previous post, “Accessibility of municipal websites in Norway after Web Accessibility Directive – statements analysis“, 11 municipalities claimed 100% conformance to Web Accessibility Directive. Running automatic accessibility tests on those 11 websites, well actually just their homepages, showed that only 2 actually didn’t have automatically detectable WCAG failures.
Only 2 out of 11 websites didn’t have automatically detectable WCAG failures, although they claimed 100% conformance.
Simply checking their homepages with automatic accessibility tests revealed WCAG problems.
Municipality | Failed WCAG Success Criterions |
---|---|
Municipality nr.1 |
|
Municipality nr.2 |
|
Municipality nr.3 |
|
Municipality nr.4 |
|
Municipality nr.5 |
|
Municipality nr.6 |
|
Municipality nr.7 |
|
Municipality nr.8 |
|
Municipality nr.9 |
|
Automatic testing has it’s limitations, sometimes even reports false positives and needs human interpretation at this time (we all hope that some day machine learning and artificial intelligence will help, but it’s still too early considering the complexity and heavy context dependence).
Still we should use automatic testing as a supplement method and especially when dealing with larger sites and sites that are updated often. They can prevent some obvious accessibility problems and help with improving overall accessibility when used wisely.
It is obvious that the data from our automatic tests and claims made by some municipalities don’t match and I will dig into details in one of the following posts.
Please check out part four where I analyze way beyond homepages, actually I analyzed 17837 URLs belonging to different municipalities – “Accessibility of municipal websites in Norway after Web Accessibility Directive – more on automatic accessibility tests“.