This is the second part of my analysis. In case you missed it – please check the first part – introduction to the accessibility of municipal websites in Norway after WAD for motivation, methodology and preparation.
Skip to accessibility statements analysis.
Structured data makes analysis easy even if we don’t have direct access to database
Analyzing unstructured data can be difficult and full of manual tweaking. I did some trivial machine learning sentiment analysis in the past and got quite good results. But sentiment analysis isn’t so hard compared to processing unstructured accessibility statements. I am confident that somebody more experienced in natural language processing analysis could get quite far, together with reinforced learning, but for me it would probably mean I would not be ready to publish this in months.
But processing structured data isn’t very difficult, especially when data is in HTML form and based on same templates. Then we only need to check the structure and relationships of HTML and CSS and we can very easily process a lot of pages. This was also the case for this analysis of mine – where all accessibility statements for public sector in Norway use the same HTML template as they are required to be a part of centralized solution called uustatus.no (in Norwegian, opens in new window).
Authorities have of course the advantage of querying their database directly and if the template is changed I need to make changes in my scripts as well. But my intention is not to check the data every day, so it’s totally doable. Another problem I had was that some municipality websites didn’t have the link to uustatus on their homepages. Often I needed to manually check the site and use their search and chat-bot solutions to come to the link I wanted to analyze – the uustatus based accessibility statement.
Therefore I call this data collection semi-automatic as I was able to get all the statements automatically in approximately 90% of all cases.
Accessibility statement analysis
There are 356 municipalities in Norway at the time I did the analysis (between 6th and 10th of April 2023). I was only able to find 296 accessibility statements published on uustatus.no. This basically means that 60 municipalities passed the deadline that was 1st of February 2023 (set by authorities).
Found and processed accessibility statement | Unable to find accessibility statement |
296 | 60 |
Objective clarification is in place – I wasn’t able to find accessibility statements for 60 municipalities, but that doesn’t mean that they didn’t made them. It only means that they didn’t published them and that I wasn’t able to find them by search on page. This is in my interpretation the same as if they didn’t have them, together with some pages that didn’t use the required uustatus solution for accessibility statement.
Authorities have again total overview about how many municipalities still didn’t fill out the required uustatus accessibility statements and it seems that municipalities aren’t the only one late.
More than 2100 public sector organizations filled out their accessibility statements but there are still more than 550 that are missing it.
Norwegian authorities about 550 public sector organizations that missed the deadline of 1st of February 2023 (in Norwegian, opens in new window).
How accessible are websites of Norwegian municipalities according to themselves
Before we dig into numbers let’s make one thing clear – accessibility statements are produced by employees of municipalities with some training and often they got external help from accessibility professionals.
I also suspect that their technical providers (Content Management System) had to evaluate their own parts (I would be surprised if that was not the case) and end product – accessibility statement can therefore vary in it’s data quality.
When I did automatic and manual tests (please check that analysis in next posts) I noticed differences and that is the reality. Accessibility is a wide and deep subject and it takes time and training to be objective even if Web Content Accessibility Guidelines (WCAG) try to give you the tools to do it.
11 out of 296 municipalities stated that they don’t have any WCAG 2.1 failures (on levels A and AA). This is a very bold statement and one of my goals was to check if that was true. In one of the next posts I will dive into details and show the data that will talk for itself.
11 out of 296 municipalities state in their accessibility statements that they don’t have any WCAG 2.1 A or AA failures on their websites.
After I processed all 296 accessibility statements I got the 11 that stated they totally conform to WCAG 2.1 on A and AA.
All 285 municipalities (296 – 11) noted 2375 problems with different success criteria. This means that every website (not webpage, but as a whole site) for a municipality have on average 8.33 problematic WCAG 2.1 success criteria on levels A and AA.
WCAG Success Criterion | # |
---|---|
1.1.1 Non-text Content (A) | 170 |
1.3.1 Info and Relationships (A) | 161 |
4.1.1 Parsing (A) | 157 |
2.4.4 Link Purpose (In Context) (A) | 149 |
1.2.2 Captions (Prerecorded) (A) | 125 |
4.1.2 Name, Role, Value (A) | 122 |
1.4.3 Contrast (Minimum) (AA) | 109 |
1.4.5 Images of Text (AA) | 109 |
1.2.1 Audio-only and Video-only (Prerecorded) (A) | 77 |
2.4.6 Headings and Labels (AA) | 71 |
1.3.5 Identify Input Purpose (AA) | 70 |
2.4.1 Bypass Blocks (A) | 66 |
2.5.3 Label in Name (A) | 64 |
3.1.2 Language of Parts (AA) | 61 |
1.4.11 Non-text Contrast (AA) | 60 |
4.1.3 Status Messages (AA) | 57 |
2.1.1 Keyboard (A) | 56 |
1.4.1 Use of Color (A) | 47 |
2.4.7 Focus Visible (AA) | 47 |
3.3.2 Labels or Instructions (A) | 47 |
3.3.1 Error Identification (A) | 45 |
1.4.4 Resize text (AA) | 43 |
3.2.1 On Focus (A) | 42 |
3.3.3 Error Suggestion (AA) | 42 |
1.3.2 Meaningful Sequence (A) | 41 |
3.1.1 Language of Page (A) | 39 |
1.4.10 Reflow (AA) | 38 |
1.4.12 Text Spacing (AA) | 36 |
1.3.3 Sensory Characteristics (A) | 28 |
2.4.2 Page Titled (A) | 26 |
2.1.2 No Keyboard Trap (A) | 21 |
2.4.3 Focus Order (A) | 20 |
3.3.4 Error Prevention (Legal, Financial, Data) (AA) | 19 |
2.2.1 Timing Adjustable (A) | 15 |
1.4.13 Content on Hover or Focus (AA) | 15 |
2.1.4 Character Key Shortcuts (A) | 12 |
2.5.4 Motion Actuation (A) | 11 |
3.2.4 Consistent Identification (AA) | 9 |
2.4.5 Multiple Ways (AA) | 8 |
2.2.2 Pause, Stop, Hide (A) | 8 |
2.5.2 Pointer Cancellation (A) | 7 |
3.2.3 Consistent Navigation (AA) | 7 |
2.5.1 Pointer Gestures (A) | 6 |
1.4.2 Audio Control (A) | 4 |
1.3.4 Orientation (AA) | 4 |
3.2.2 On Input (A) | 2 |
2.3.1 Three Flashes or Below Threshold (A) | 2 |
As mentioned in the beginning, self-declared WCAG problems may be different from reality. It would take a lot of resources to actually check everything manually and with time data will for sure be more relevant. For this first analysis after Web Accessibility Directive we can anyway be satisfied with reports that find more issues than automatic testing alone can.
Some interesting founds
When I went through pages, looking for correct accessibility statements (only those on uustatus.no are correct after 1st of February 2023) I happened to find some interesting facts;
- Some chat-bots were unable to find accessibility statements and offered links from authority instead. They could at least make a small effort and be opened about their work in progress.
- Couple of municipalities offered their accessibility statements as PDF file, itself not tagged and therefore inaccessible.
- Some municipalities referred to automatic test scoring – like “our site reached 65 out of 100 points” and the name of the automatic accessibility testing provider/tool.
- I was not able to find any accessibility overlays on sites I had to check manually (at least 10 % of all tested), which is a good thing.
- I was able to find an old accessibility statement that was referring to WCAG version 1.
- Some municipalities were really documenting their overall accessibility and I found one that referred to 50 accessibility statements (39 of them of third parties).
Statements are one thing, but can we also check something else? Sure we can, the fastest way is to use automatic accessibility tests and that is what you can read about in part 3.