Accessibility scoring don’t tell much about actual state of accessibility

With Web Accessibility Directive (WAD) now more than two years old we probably see more accessibility statements when we browse, especially on public sector websites in European Union.

I like that and I also like that sometimes even private sector tries to make their accessibility statements (especially those companies that are often selling to private sector). Another positive impact of WAD that can be seen as an improvement.

Because it’s not necessarily that. I have to write about different scores that sometimes appear in accessibility statements and are actually not presenting the whole truth about accessibility although their name says exactly that they are.

Your automatic test is saying your accessibility score is 100% and you can wrongly think you conform to WCAG

I’ve seen folks using Google Lighthouse accessibility score on their accessibility statement as a proof that their site is compliant with WCAG 2.1 on levels A and AA. A lot of larger businesses like to use some corporate tooling scoring that mentions their accessibility score is 100/100 and similar.

I like automatic tools, don’t get me wrong. I even coded my own automatic accessibility testing tool – aXeSiA, just to see how I can get a better statistic analysis based on all public facing sites on a domain.

They enable us with bulk test results, to catch the problems that can be easy to detect in the code and sometimes also in the interpreted code (like CSS and HTML combined).

But let me remind us about some major downsides of such tools:

1. Automatic tools can only cover up to ca. 30% of all WCAG success criteria.
2. Automatic tools can only detect fails but often they can’t confirm passes.
3. Automatic tools can sometimes report false positives – fails that are actually not fails
4. Automatic tools can sometimes fail to check the site – they are prone to bugs as any other software

With these downsides in mind we can quickly conclude that even when we get a 100% pass from an automatic tool that doesn’t really mean a lot in regards with WCAG. It means that our code seems to be without detectable faults.

So writing that your score is 100% doesn’t actually mean a lot. It just means that your code is passing some predefined test rules that can’t really cover everything WCAG does.

Please try to understand that when you use your score as proof. It’s a good thing to pass all those automatic tests, I am not arguing that. But it is far from enough and it’s only a start, not the end. It’s a good start, don’t get me wrong, but it is only a start and actually a minimum.

Dirty details behind automatic accessibility testing

Automatic accessibility testing tools try to cover as much of WCAG success criteria as possible. I hope that some day we can get to a point that they are so reliable that we can trust them 100% and concentrate on user testing. But it will take a lot of time and effort if it will even be possible. Call me an optimistic realist. For now, we are quite behind and it seems like it will always be like that, considering the efforts made on progress overall and progress of automatic accessibility testing.

Automatic testing can prevent releases of code that is most likely to negatively effect overall accessibility. But they can’t really confirm that the the code really makes our product accessible. They can only find fails, but they can’t confirm passes – a simple example is if image is missing alt attribute. That’s a 100% fail. But if it has the attribute and it is empty it doesn’t mean that it is 100% correct pass. The image may not be decorative, so alt text should be there.

As Manuel Matuzović wrote in his “Building the most inaccessible site possible with a perfect Lighthouse score” (opens in new window) – we can very quickly make a totally inaccessible website that still gets the best scores when checked with automatic accessibility testing tools.

And it isn’t really strange when we know about the background.

Almost all accessibility testing tools are based on similar or even same test rules, defined by accessibility professionals gathered in ACT Rules Community (W3C group, opens in new window). ACT Rules community work on testable rules (opens in new window) that can then be implemented as automatic tests in different tools, like Deque’s axe and Google’s Lighthouse (which is actually using axe behind the scenes (opens in new window)).

So at the time of writing this blog post there are 91 ACT rules (including deprecated). This does not mean that they cover all different WCAG success criteria! I did a quick analysis and parsed them so that I can capture current state for you.

ACT rules define 38 out of 78 WCAG success criterion tests for fails

After parsing the list of ACT rules I managed to discover that there are 38 rules (or tests) that can be mapped to WCAG success criteria. This means that the group managed to write 38 rules out of 78 (all WCAG 2.1 success criteria on all levels (A, AA and AAA)).

1. 1.1.1 Non-Text Content (Level A)
2. 1.2.1 Audio-Only And Video-Only (Prerecorded) (Level A)
3. 1.2.2 Captions (Prerecorded) (Level A)
4. 1.2.3 Audio Description Or Media Alternative (Prerecorded) (Level A)
5. 1.2.5 Audio Description (Prerecorded) (Level AA)
6. 1.2.8 Media Alternative (Prerecorded) (Level AAA)
7. 1.3.1 Info And Relationships (Level A)
8. 1.3.3 Sensory Characteristics (Level A)
9. 1.3.4 Orientation (Level AA)
10. 1.3.5 Identify Input Purpose (Level AA)
11. 1.4.12 Text Spacing (Level AA)
12. 1.4.2 Audio Control (Level A)
13. 1.4.3 Contrast (Minimum) (Level AA)
14. 1.4.4 Resize Text (Level AA)
15. 1.4.5 Images Of Text (Level AA)
16. 1.4.6 Contrast (Enhanced) (Level AAA)
17. 1.4.9 Images Of Text (No Exception) (Level AAA)
18. 2.1.1 Keyboard (Level A)
19. 2.1.2 No Keyboard Trap (Level A)
20. 2.1.3 Keyboard (No Exception) (Level AAA)
21. 2.1.4 Character Key Shortcuts (Level A)
22. 2.2.1 Timing Adjustable (Level A)
23. 2.2.2 Pause, Stop, Hide (Level A)
24. 2.2.4 Interruptions (Level AAA)
25. 2.4.1 Bypass Blocks (Level A)
26. 2.4.2 Page Titled (Level A)
27. 2.4.4 Link Purpose (In Context) (Level A)
28. 2.4.6 Headings And Labels (Level AA)
29. 2.4.7 Focus Visible (Level AA)
30. 2.4.9 Link Purpose (Link Only) (Level AAA)
31. 2.5.3 Label In Name (Level A)
32. 2.5.4 Motion Actuation (Level A)
33. 3.1.1 Language Of Page (Level A)
34. 3.1.2 Language Of Parts (Level AA)
35. 3.2.5 Change On Request (Level AAA)
36. 3.3.1 Error Identification (Level A)
37. 4.1.1 Parsing (Level A)
38. 4.1.2 Name, Role, Value (Level A)

Not all of them are actually implemented by automatic accessibility testing tools, but the rules are defined and potentially it means that they will also be implemented.

Now there is a problem – as mentioned before – the rules don’t cover passes but only fails, and also that is not the whole story – the fails are limited to some of multiple possible.

So – again – please don’t think that automatic accessibility testing tools can now find if you conform to half of whole WCAG 2.1! This only means that there are theoretical rules that can find fails for 38 of 78 WCAG success criteria.

Automatic accessibility testing tools can only find failures. Passes can only be established manually. Watch also out for falsely reported failures, all software has bugs and border cases.

my reflection about automatic accessibility testing tools.

There are 101 accessibility requirement mappings for 88 non-deprecated rules

Some rules are deprecated, so I removed them and out of 91 rules 88 are non-deprecated. Rules can cover multiple accessibility requirements mappings, so this means that 1 rule can cover 1 WCAG and 1 best practice, for example.

Number	Accessibility Requirements Mapping	Is WCAG Success Criterion?	Rules
1	This rule is not required for conformance.	false	ARIA attribute is defined in WAI-ARIA, Audio element content has transcript, Audio element content is media alternative for text, Document has a landmark with non-repeated content, Element marked as decorative is not exposed, Focusable element has no keyboard trap via non-standard navigation, Focusable element has no keyboard trap via standard navigation, Video element content is media alternative for text, Video element visual-only content is media alternative for text
2	1.3.1 Info And Relationships (Level A)	true	ARIA required context role, ARIA required owned elements, Headers attribute specified on a cell refers to cells in the same table element, Heading has non-empty accessible name, Role attribute has valid value, Table header cell has assigned cells
3	ARIA 1.1, 6.3 Values For States And Properties	false	ARIA state or property has valid value
4	ARIA5: Using WAI-ARIA State And Property Attributes To Expose The State Of A User Interface Component	false	ARIA state or property is permitted, Element with role attribute has required states and properties
5	ARIA 1.1, 7.6 State And Property Attribute Processing	false	ARIA state or property is permitted
6	4.1.1 Parsing (Level A)	true	Attribute is not duplicated, Id attribute value is unique
7	H94: Ensuring That Elements Do Not Contain Duplicate Attributes	false	Attribute is not duplicated
8	1.2.8 Media Alternative (Prerecorded) (Level AAA)	true	Audio and visuals of video element have transcript, Video element visual content has accessible alternative
9	G69: Providing An Alternative For Time Based Media	false	Audio and visuals of video element have transcript, Video element visual content has accessible alternative
10	1.2.1 Audio-Only And Video-Only (Prerecorded) (Level A)	true	Audio element content has text alternative, Video element visual-only content has accessible alternative
11	G158: Providing An Alternative For Time-Based Media For Audio-Only Content	false	Audio element content has text alternative
12	Audio Element Content Has Transcript	false	Audio element content has text alternative
13	Audio Element Content Is Media Alternative For Text	false	Audio element content has text alternative
14	1.4.2 Audio Control (Level A)	true	Audio or video element avoids automatically playing audio
15	WCAG Non-Interference	false	Audio or video element avoids automatically playing audio, Focusable element has no keyboard trap, Text content that changes automatically can be paused, stopped or hidden
16	G60: Playing A Sound That Turns Off Automatically Within Three Seconds	false	Audio or video element avoids automatically playing audio, Audio or video element that plays automatically has no audio that lasts more than 3 seconds
17	G170: Providing A Control Near The Beginning Of The Web Page That Turns Off Sounds That Play Automatically	false	Audio or video element avoids automatically playing audio, Audio or video element that plays automatically has a control mechanism
18	G171: Playing Sounds Only On User Request	false	Audio or video element avoids automatically playing audio
19	Audio Or Video Element That Plays Automatically Has A Control Mechanism	false	Audio or video element avoids automatically playing audio
20	Audio Or Video Element That Plays Automatically Has No Audio That Lasts More Than 3 Seconds	false	Audio or video element avoids automatically playing audio
21	1.3.5 Identify Input Purpose (Level AA)	true	Autocomplete attribute has valid value
22	SCR28: Using An Expandable And Collapsible Menu To Bypass Block Of Content	false	Block of repeated content is collapsible, Bypass Blocks of Repeated Content
23	4.1.2 Name, Role, Value (Level A)	true	Button has non-empty accessible name, Element with aria-hidden has no content in sequential focus navigation, Element with presentational children has no focusable content, Element with role attribute has required states and properties, Form field has non-empty accessible name, Iframe element has non-empty accessible name, Iframe elements with identical accessible names have equivalent purpose, Image button has non-empty accessible name, Link has non-empty accessible name, Menuitem has non-empty accessible name
24	2.4.1 Bypass Blocks (Level A)	true	Bypass Blocks of Repeated Content
25	G1: Adding A Link At The Top Of Each Page That Goes Directly To The Main Content Area	false	Bypass Blocks of Repeated Content, Document has an instrument to move focus to non-repeated content
26	G123: Adding A Link At The Beginning Of A Block Of Repeated Content To Go To The End Of The Block	false	Bypass Blocks of Repeated Content, Document has an instrument to move focus to non-repeated content
27	G124: Adding Links At The Top Of The Page To Each Area Of The Content	false	Bypass Blocks of Repeated Content, Document has an instrument to move focus to non-repeated content
28	H69: Providing Heading Elements At The Beginning Of Each Section Of Content	false	Bypass Blocks of Repeated Content, Document has heading for non-repeated content
29	Document Has Heading For Non-Repeated Content	false	Bypass Blocks of Repeated Content
30	Document Has A Landmark With Non-Repeated Content	false	Bypass Blocks of Repeated Content
31	Block Of Repeated Content Is Collapsible	false	Bypass Blocks of Repeated Content
32	Document Has An Instrument To Move Focus To Non-Repeated Content	false	Bypass Blocks of Repeated Content
33	1.3.3 Sensory Characteristics (Level A)	true	Content has alternative for visual reference
34	G96: Providing Textual Identification Of Items That Otherwise Rely Only On Sensory Information To Be Understood	false	Content has alternative for visual reference
35	2.5.4 Motion Actuation (Level A)	true	Device motion based changes to the content can also be created from the user interface, Device motion based changes to the content can be disabled
36	2.4.7 Focus Visible (Level AA)	true	Element in sequential focus order has visible focus
37	Fourth Rule Of ARIA Use	false	Element with aria-hidden has no content in sequential focus navigation
38	3.1.2 Language Of Parts (Level AA)	true	Element with lang attribute has valid language tag, HTML element language subtag matches language
39	H58: Using Language Attributes To Identify Changes In The Human Language	false	Element with lang attribute has valid language tag, HTML element language subtag matches language
40	3.3.1 Error Identification (Level A)	true	Error message describes invalid form field value
41	2.1.2 No Keyboard Trap (Level A)	true	Focusable element has no keyboard trap
42	G21: Ensuring That Users Are Not Trapped In Content	false	Focusable element has no keyboard trap
43	Focusable Element Has No Keyboard Trap Via Standard Navigation	false	Focusable element has no keyboard trap
44	Focusable Element Has No Keyboard Trap Via Non-Standard Navigation	false	Focusable element has no keyboard trap
45	2.4.6 Headings And Labels (Level AA)	true	Form field label is descriptive, Heading is descriptive
46	G131: Providing Descriptive Labels	false	Form field label is descriptive
47	H43: Using Id And Headers Attributes To Associate Data Cells With Header Cells In Data Tables	false	Headers attribute specified on a cell refers to cells in the same table element
48	H42: Using H1-H6 To Identify Headings	false	Heading has non-empty accessible name
49	1.4.5 Images Of Text (Level AA)	true	HTML graphics contain no text
50	1.4.9 Images Of Text (No Exception) (Level AAA)	true	HTML graphics contain no text
51	3.1.1 Language Of Page (Level A)	true	HTML page has lang attribute, HTML page lang and xml:lang attributes have matching values, HTML page lang attribute has valid language tag, HTML page language subtag matches default language
52	H57: Using Language Attributes On The Html Element	false	HTML page has lang attribute, HTML page lang attribute has valid language tag, HTML page language subtag matches default language
53	2.4.2 Page Titled (Level A)	true	HTML page has non-empty title, HTML page title is descriptive
54	G88: Providing Descriptive Titles For Web Pages	false	HTML page has non-empty title, HTML page title is descriptive
55	H25: Providing A Title Using The Title Element	false	HTML page has non-empty title, HTML page title is descriptive
56	H93: Ensuring That Id Attributes Are Unique On A Web Page	false	Id attribute value is unique
57	2.1.1 Keyboard (Level A)	true	Iframe with negative tabindex has no interactive elements, Scrollable element is keyboard accessible
58	G202: Ensuring Keyboard Control For All Functionality	false	Iframe with negative tabindex has no interactive elements, Scrollable element is keyboard accessible
59	1.1.1 Non-Text Content (Level A)	true	Image accessible name is descriptive, Image button has non-empty accessible name, Image has non-empty accessible name, Image not in the accessibility tree is decorative, Object element rendering non-text content has non-empty accessible name, SVG element with explicit role has non-empty accessible name
60	G94: Providing Short Text Alternative For Non-Text Content That Serves The Same Purpose And Presents The Same Information As The Non-Text Content	false	Image accessible name is descriptive, Image button has non-empty accessible name, Image has non-empty accessible name
61	G95: Providing Short Text Alternatives That Provide A Brief Description Of The Non-Text Content	false	Image accessible name is descriptive, Image button has non-empty accessible name, Image has non-empty accessible name
62	1.4.12 Text Spacing (Level AA)	true	Letter spacing in style attributes is not !important, Line height in style attributes is not !important, Word spacing in style attributes is not !important
63	2.4.4 Link Purpose (In Context) (Level A)	true	Link has non-empty accessible name, Link in context is descriptive, Links with identical accessible names and same context serve equivalent purpose
64	2.4.9 Link Purpose (Link Only) (Level AAA)	true	Link has non-empty accessible name, Link in context is descriptive, Link is descriptive, Links with identical accessible names and same context serve equivalent purpose, Links with identical accessible names have equivalent purpose
65	G91: Providing Link Text That Describes The Purpose Of A Link	false	Link has non-empty accessible name
66	2.2.1 Timing Adjustable (Level A)	true	Meta element has no refresh delay
67	2.2.4 Interruptions (Level AAA)	true	Meta element has no refresh delay, Meta element has no refresh delay (no exception)
68	3.2.5 Change On Request (Level AAA)	true	Meta element has no refresh delay, Meta element has no refresh delay (no exception)
69	G110: Using An Instant Client-Side Redirect	false	Meta element has no refresh delay, Meta element has no refresh delay (no exception)
70	H76: Using Meta Refresh To Create An Instant Client-Side Redirect	false	Meta element has no refresh delay, Meta element has no refresh delay (no exception)
71	1.4.4 Resize Text (Level AA)	true	Meta viewport allows for zoom, Zoomed text node is not clipped with CSS overflow
72	2.1.4 Character Key Shortcuts (Level A)	true	No keyboard shortcut uses only printable characters
73	1.3.4 Orientation (Level AA)	true	Orientation of the page is not restricted using CSS transform property
74	ARIA4: Using A WAI-ARIA Role To Expose The Role Of A User Interface Component	false	Role attribute has valid value
75	G108: Using Markup Features To Expose The Name And Role, Allow User-Settable Properties To Be Directly Set, And Provide Notification Of Changes	false	Role attribute has valid value
76	2.1.3 Keyboard (No Exception) (Level AAA)	true	Scrollable element is keyboard accessible
77	2.2.2 Pause, Stop, Hide (Level A)	true	Text content that changes automatically can be paused, stopped or hidden
78	1.4.6 Contrast (Enhanced) (Level AAA)	true	Text has enhanced contrast, Text has minimum contrast
79	G17: Ensuring That A Contrast Ratio Of At Least 7:1 Exists Between Text (And Images Of Text) And Background Behind The Text	false	Text has enhanced contrast
80	G18: Ensuring That A Contrast Ratio Of At Least 4.5:1 Exists Between Text (And Images Of Text) And Background Behind The Text	false	Text has enhanced contrast
81	1.4.3 Contrast (Minimum) (Level AA)	true	Text has minimum contrast
82	1.2.2 Captions (Prerecorded) (Level A)	true	Video element auditory content has accessible alternative
83	G87: Providing Closed Captions	false	Video element auditory content has accessible alternative, Video element auditory content has captions
84	G93: Providing Open (Always Visible) Captions	false	Video element auditory content has accessible alternative, Video element auditory content has captions
85	H95: Using The Track Element To Provide Captions	false	Video element auditory content has accessible alternative, Video element auditory content has captions
86	Video Element Content Is Media Alternative For Text	false	Video element auditory content has accessible alternative, Video element visual content has accessible alternative, Video element visual content has strict accessible alternative
87	Video Element Auditory Content Has Captions	false	Video element auditory content has accessible alternative
88	1.2.3 Audio Description Or Media Alternative (Prerecorded) (Level A)	true	Video element visual content has accessible alternative
89	1.2.5 Audio Description (Prerecorded) (Level AA)	true	Video element visual content has accessible alternative, Video element visual content has strict accessible alternative
90	G8: Providing A Movie With Extended Audio Descriptions	false	Video element visual content has accessible alternative, Video element visual content has audio description, Video element visual content has strict accessible alternative
91	G78: Providing A Second, User-Selectable, Audio Track That Includes Audio Descriptions	false	Video element visual content has accessible alternative, Video element visual content has audio description, Video element visual content has strict accessible alternative
92	G173: Providing A Version Of A Movie With Audio Descriptions	false	Video element visual content has accessible alternative, Video element visual content has audio description, Video element visual content has strict accessible alternative
93	Video Element Visual Content Has Audio Description	false	Video element visual content has accessible alternative, Video element visual content has strict accessible alternative
94	Audio And Visuals Of Video Element Have Transcript	false	Video element visual content has accessible alternative
95	G159: Providing An Alternative For Time-Based Media For Video-Only Content	false	Video element visual-only content has accessible alternative, Video element visual-only content has transcript
96	G166: Providing Audio That Describes The Important Video Content And Describing It As Such	false	Video element visual-only content has accessible alternative, Video element visual-only content has audio track alternative
97	Video Element Visual-Only Content Is Media Alternative For Text	false	Video element visual-only content has accessible alternative
98	Video Element Visual-Only Content Has Transcript	false	Video element visual-only content has accessible alternative
99	Video Element Visual-Only Content Has Audio Track Alternative	false	Video element visual-only content has accessible alternative
100	2.5.3 Label In Name (Level A)	true	Visible label is part of accessible name
101	G208: Including The Text Of The Visible Label As Part Of The Accessible Name	false	Visible label is part of accessible name

To summarize – automatic testing alone is far from enough and can’t tell the real state

If you have read so far, I hope you have a bit better understanding about relevance behind automatic testing and scoring based on that. I can’t repeat this enough – automatic testing often needs manual validation. So don’t trust it totally and verify the results. It is of course very nice to be able to scan hundreds or even thousands of URLs very quickly, but the reported results are only the beginning.

Automatic results needs manual verification and therefore automatic scoring can’t really be trusted. Once again – perfect score on problematic websites is still a reality and we need human specialists and people with disabilities to really detect the real situation.