WebAIM recently published latest (fifth) report of their automatic accessibility evaluation called WebAIM Million (opens in new window). It is the largest publicly shared accessibility evaluation known to me and it is extremely interesting for me to check it out as I can compare it with my experiences when doing manual and automatic accessibility testing.
In this post I will reflect on some points that are maybe not so obvious.
3.7% of evaluated pages didn’t have (automatically detectable) WCAG failures
If we compare this with previous WebAIM Million from 2022 (opens in new window), we get an indication of extremely small improvement. In 2022 96.8% of pages had detectable WCAG failures and in 2023 “only” 96.3% had failures.
We must understand that automatic testing for accessibility can’t really pass WCAG criteria, except in some parts. Therefore we can’t say that 3.7% out of the million pages tested are conforming to WCAG. And we also can’t say that 3.7% of pages are accessible considering the limitations of automatic accessibility testing and also WCAG itself.
Report isn’t actually telling us that. I like the honesty and professionalism of the report where they are explaining that correctly.
These are only automatically detected errors that align with WCAG conformance failures with a high level of reliability. Because automatic testing cannot detect all possible WCAG failure types, this means that the actual WCAG 2 A/AA conformance level was certainly lower.
WebAIM Million on WCAG conformance (opens in new window).
But I have seen a lot of interpretations of the report that try to pass the ratio as a fact. So please if you detect that somebody is interpreting the report is actually saying that 3.7% of websites are conforming or even accessible try to remember the facts behind the methodology.
We can conclude with pretty high certainty that 96.3% of pages have detectable WCAG 2 failures and most probably the rest have undetectable failures as well. Especially when we consider two facts – the maximum WCAG coverage of automatic tests and that automatic accessibility testing can sometimes also return false positives. To understand more about the difference between automatic and manual accessibility testing check out Adrian Roselli’s comparison (opens in new window).
I still love the WebAIMs project, don’t get me wrong. Testing 37000 (that is 3.7% of a million) webpages manually would be required to establish the reality. That would mean quite a funding and/or extremely wide network of accessibility specialists volunteering. So I respect the WebAIMs results and treat them as an indication.
After all – without data we are just people with opinions.
Without data, you’re just another person with an opinion.
Wiliam Edwards Deming
Accessibility failure trends from 2019 to 2023 according to WebAIMs Million
I have seen the graphs on the page with trends from 2019 to 2023, but I missed a table that would include the main points for all years, so here you go:
| Year | % WCAG failures | # errors per page | # of elements |
|---|---|---|---|
| 2023 | 96.3 | 50.0 | 1050 |
| 2022 | 96.8 | 50.8 | 955 |
| 2021 | 97.4 | 51.4 | 887 |
| 2020 | 98.1 | 60.9 | 864 |
| 2019 | 97.8 | 59.6 | 782 |
It’s obvious that detectable failures are now finally improving, but it’s interesting that the page complexity is rising.
Page complexity trends may mislead
Page complexity in the report is measured by number of elements and in my opinion isn’t really the indication of complexity that matters for accessibility, as we don’t know if elements bear semantic meaning or not. If they don’t bear any semantics or are not important to accessibility (not impacting the accessibility tree), then they don’t really reflect correct complexity in my opinion.
Correct complexity in the lights of accessibility would for me mean counting elements that are important for accessibility tree. So if we have hundreds of div or span elements with no special CSS or ARIA on it – they would be mostly ignored by assistive technologies. There are for sure some negative impacts on the performance but as far as I know hundred div or span elements don’t have detectable impact.
I am still waiting for clarifications from Google Lighthouse on my proof of concept to measure performance that matters for accessibility, but no comments from October 2022.
If I was in charge of the project I would suggest that they at least added weights to elements with semantic meaning. I was using some weights in my open source automatic accessibility testing tool aXeSiA and they did quite a good job pointing to pages with most complexity.
Presence of ARIA seems to negatively impact accessibility
Report mentions that ARIA does not necessarily introduce more errors, but it is a fact that there are more errors present on pages that use ARIA. I think this is again related with the possibilities of automatic accessibility testing. ARIA has a lot of code-based rules and those can be quite easily checked with automatic tools. Therefore tools can also find more accessibility issues as if we don’t use ARIA at all.
Increased ARIA usage on pages was associated with higher detected errors. The more ARIA attributes that were present, the more detected accessibility errors could be expected
WebAIM Million on ARIA (opens in new window).
This does not necessarily mean that ARIA introduced these errors (these pages are more complex), but pages typically had more errors when ARIA was present.
I think that report makes this quite clear as well, but I remember that some people like to use this to promote lesser usage of ARIA. This is quite dangerous as well – as ARIA is critical in some cases, especially in providing semantics that isn’t there in the raw HTML. Sure, there are many rules developers need to obey, as it is again true that with great power we also get great responsibility.
Technologies seem to impact accessibility
If we don’t really read the text around tables provided by the report we may establish that Bootstrap, WordPress, Vue.js, jQueryUI, Laravel, Woocommerce and others worsen accessibility.
The report makes it quite clearly that it isn’t the case. But again, we need to be careful with interpretation. Basics for accessibility is proper HTML, CSS and JavaScript and some parts of some technologies do make it simpler or harder for us, yes, but on the end we as developers need to know what will be rendered to browser and with that to assistive technologies.
The responsibility is ours – developers must know how to use the tools to generate accessible code. Even no-code or low-code tools have possibilities to produce accessible rendering code. If they don’t we have to report bugs or if that isn’t an option choose other tools. It again drills down to training and knowledge.
Most common errors have been the same for the last 5 years
96.1% of all errors detected fall into six categories:
- Low contrast text – .3% better compared to 2022
- Missing alternative text for images – 2.8% worse compared to 2022
- Empty links – 0.4% worse compared to 2022
- Missing form input labels – 0.2% better compared to 2022
- Empty buttons – 0.3% better compared to 2022
- Missing document language – 3.7 better compared to 2022
Report states that these most common errors have been the same for the last 5 years. I don’t think that is very surprising because automatic tools are actually best in detecting exactly these WCAG errors. So the approximate 30% of all automatically detectable WCAG failures will normally fall into these failures and the rest (let’ say 70%) will not be detected because they are only detectable by manual testing.
The fact still remains – all six errors could be very simply prevented with just basic accessibility training and processes that would prevent such errors being released. It would be extremely interesting to know about the changes on the sites that were tested. For example – if 50% of all pages tested were re-made and we still have the same issues and similar. I don’t think that would be an easy task but it WebAIM is saving the HTML together with results it could be done quite easily.
If same errors are there because the page didn’t change is one thing. If same errors are there even after page was totally renewed that is another thing. First problem – legacy code – can indicate other issues and the second problem indicates that people responsible need proper training and understanding. It also indicates other problems, I will guess, in procedures, planning, procurement and so on. But it again drills down to knowledge and processes.
Some thoughts about reasons and solutions
There are a lot of facts in the report that all point to single problem and solution, in my opinion. It is the lack of training, knowledge and responsibility (all three are somehow the same thing I would like to think). Another example is that only 16.8% of all HTML tables are coded correctly with valid data markup.
I also agree with Eric Eggert on most of his points made in We need accessibility action post (opens in new window). Among all suggestions he made there I love the knowledge/training part the most;
because fixing those six issues can be learned in a day
Eric Eggert’s We need accessibility action post (opens in new window).
It really is so simple and at the same time it is also an indication about the main reason behind the problems we have. Lack of understanding and knowledge. Even schools that are supposed to teach don’t get it right, so we really need action, yes, on all levels and for all roles.
