Automated accessibility testing: what a preregistered 285-page scan actually shows

TL;DR: We scanned 285 public homepages sampled systematically across the full Tranco top-1M rank range. 89.5% of them (95% CI 85.4 to 92.5) failed at least one automatically detectable WCAG 2.0/2.1 Level A or AA check, and that held under every counting convention we preregistered. Our second hypothesis, that a handful of failure types account for the bulk of all errors, did not hold: two of our three preregistered robustness checks falsified it. We are publishing both, because how much of accessibility a scanner appears to catch depends enormously on what you decide to count.

What we ran, and how

This is a systematic structural measurement of live public homepages. It is not a comprehensive study of web accessibility, not a user experience survey, and not an audit of whether real people can use these sites. That scope applies to every number below.

We sampled from the Tranco top-1M list 64YYX, created 2026-07-19. Tranco is the research standard because it is manipulation resistant and every daily list has a permanent citable ID, so anyone can rebuild our exact sample frame. We took every 2,500th rank from offset 1,250, spreading the sample across the whole 1 to 1,000,000 range rather than concentrating it on the most resourced sites on earth.

Of 396 domains attempted, 111 were excluded (76 navigation failures, 23 HTTP 403 responses, the rest a scatter of other codes). That left N = 285 homepages scanned on 2026-07-20 with axe-core 4.12.1 driven by Playwright Chromium headless, filtered to WCAG 2.0 and 2.1 Level A and AA tags, one request per domain. Before collection we validated the instrument against a positive control fixture containing exactly one instance of each failure type under study, and axe found all six. That matters: an empty result is a clean page, not a broken scanner.

We recorded 8,629 error instances. The mean was 30.3 per page, the median 14. The distribution is heavily right skewed, so a minority of very broken pages drags the mean well above the typical page.

Nine in ten homepages fail an automated check

255 of 285 homepages, 89.5%, had at least one machine detectable Level A or AA violation. We preregistered a falsification threshold of 80% and never came close to it. It also held under every convention tested: 89.5% counting element instances, 89.5% counting distinct rules, 96.8% including best practice rules, and 97.9% treating axe's "incomplete" results as violations. Page level failure is insensitive to counting method, because a page fails the moment it has one violation of any kind.

The top band is only 29 pages, so 93.1% is directional at best. There is no clean monotonic relationship between rank and failure rate, and we are not claiming bigger sites are worse. What the bands do show is that no band escapes: the floor across all three is 87.6%. Failure is not a small site problem.

For context, the WebAIM Million 2026 report found detected WCAG 2 failures on 95.9% of the top million homepages, up from 94.8% in 2025, averaging 56.1 errors per page. Our figures sit below both. That is not evidence the web is improving. WebAIM uses the WAVE engine and we used axe-core, and the two differ in rule sets and counting conventions, so this is engine to engine and not apples to apples. The engine difference is the first candidate explanation for the gap, and we preregistered that caveat before we had results.

Then our headline number broke

Our second hypothesis was the interesting one. WebAIM reports that six failure types account for 96% of all errors they detect. We preregistered a deliberately lower bar: that the same six would account for at least 75% of error instances in our scan. Under the primary analysis they hit 94.0%, supported. Then we ran the three sensitivity checks written and frozen before any data existed, and two of them pushed that same 94.0% below the threshold, to 70.1% counting distinct rules and 53.2% including best practice rules.

Two of three preregistered checks falsify our own headline. We are not going to dress that up as a strength, and we are not going to apologise for it either. Our preregistration said in writing that both outcomes were publishable and that a divergence would be reported as a divergence rather than reconciled away. This is us keeping that promise.

The mechanism is not mysterious. Counting element instances rewards failures that replicate: one bad contrast rule in a stylesheet can generate hundreds of individually counted instances on a single page, while a subtler structural defect appears once. Count distinct rules and the concentration drops to 70.1%. Widen beyond WCAG to axe's best practice rules and it drops to 53.2%.

The same fragility hits the biggest single category. Low contrast is 56.3% of error instances under the primary analysis, but moves to 24.3% counting rules and 31.8% including best practice rules, so we do not quote it alone. The stable form is this: low contrast was present on 67.7% of scanned pages (193 of 285), ranking first by both element instances and distinct rules no matter how you count.

One piece of reassurance survives. Remove contrast entirely and the remaining five categories still account for 86.2% of the remaining error instances. The concentration finding is not purely a contrast artifact, even though its headline percentage (94.0% primary, 70.1% by rules, 53.2% with best practice rules) clearly is convention dependent.

What automated accessibility testing does and does not cover

Here is what reconciles two facts that sound contradictory, and it is the actual insight of this post.

Measured against the WCAG success criteria, automated tooling is weak. A 2025 peer-reviewed study of Swedish public sector websites by Fischer, Lundell and Gamalielsson found that automated checking tools cover only about one sixth of all WCAG success criteria, with a median of two or fewer unique violated criteria found per tool. That is a measured finding from a study with a specific scope, not a universal constant, but it is the best sourced figure available. (If you have seen "automation catches 30 to 40% of WCAG" repeated in the industry, note that we could not trace it to any standards body, agency, or journal, so we are not printing it as measured.)

Measured against the errors actually present on real pages, automated tooling is strong. WebAIM reports six machine detectable failure types accounting for 96% of every error they detect across a million homepages, and our own scan put the same six at 94.0% of error instances (70.1% counting distinct rules, 53.2% including best practice rules).

Both are true at once because real world failures are not evenly distributed across the criteria. Most sites break the same small handful of rules, over and over, mechanically. A scanner that understands a sixth of the rulebook still meets most of the errors actually out there, because they pile up in exactly the boring, repetitive categories a machine handles well.

One caution on WebAIM's numbers, because they publish two denominators and mixing them is the easiest way to get this topic wrong. The 96% is a share of detected errors. Their per-category figures are a share of pages: low contrast 83.9%, missing alternative text 53.1%, missing form input labels 51%, empty links 46.3%, empty buttons 30.6%, missing document language 13.5%. Different measurements, not interchangeable.

What a scanner cannot tell you

The most important limitation on our dataset comes from WebAIM itself, and we would rather quote the project whose data we compare against than source this elsewhere. In their words: "All automated tools, including WAVE, have limitations, not all conformance failures can be automatically detected," and "Absence of detected errors does not indicate that a page is accessible or conformant." They add that because only automatically detectable failures were counted, true full WCAG 2 A/AA conformance was certainly lower than 4.1%.

This is not only WebAIM's position. The W3C Web Accessibility Initiative, which authors WCAG itself, is explicit in its guidance on selecting evaluation tools: tools "cannot check all accessibility aspects automatically" because "human judgement is required," they can "sometimes produce false or misleading results," and they "can not determine accessibility, they can only assist in doing so." That is the standards body itself setting the ceiling on every number in this post, ours included.

Our scan measures only machine detectable failures. It says nothing about whether a disabled person can complete a task on any of these sites. Automation can tell you an image has no alt attribute, not whether the alt text you wrote is any good. It can flag a missing form label, not judge focus order, meaningful sequence, or whether an interaction makes sense through a screen reader.

Nor is "just run a scanner in CI" solved. The same peer-reviewed work found automated tools disagree with each other, attributing that to the size, complexity and internal inconsistency of WCAG and its techniques documentation, plus ordinary defects in the tools. Which scanner you run, and how you configure it, changes the answer.

The regulator has already punished the overclaim

The FTC required overlay vendor accessiBe to pay $1 million over allegations it misrepresented its AI overlay's ability to make any website WCAG compliant, and approved the final order 3-0 on April 22, 2025. The order bars claiming an automated product can make any website compliant, or keep it compliant, without evidence. That is precisely the failure mode this post is trying to avoid: anyone arguing automated testing is undervalued, as we are, owes it to the reader not to commit the mirror image error. The stakes are not abstract either. Plaintiffs filed 3,117 website accessibility lawsuits in U.S. federal court in 2025, up 27% over 2,452 in 2024.

One deadline point needs stating precisely, because it is widely garbled. The DOJ's interim final rule of April 20, 2026 (91 FR 20902) extended ADA Title II compliance dates by one year: public entities of 50,000 or more move to April 26, 2027, smaller entities and special districts to April 26, 2028. Title II covers state and local government only. This changes nothing for private businesses under Title III. If you run a commercial website, you did not get an extension, because you never had that deadline.

Limitations

A post about measurement honesty does not get to omit these.

• Survivorship bias. 111 of 396 attempted domains were excluded, 76 via navigation failure and 23 via HTTP 403. Sites that blocked or failed our scanner are systematically unmeasured, and there is no reason to assume their accessibility resembles the measured set. The true population rate could be higher or lower.
• Collection noise. The run needed two passes. 274 pass-1 records were headless browser crashes (instrument faults, not site failures), so they were purged and re-scanned. A flaky headless browser is a documented source of collection noise.
• Deviations from the frozen preregistration. We ran at concurrency 10 rather than the preregistered 4 (still one request per domain, all different domains). We added a 2.5 second settle delay that was not preregistered, which lets more deferred content load and can raise detected error counts. We ran axe with no tag filter and applied the preregistered WCAG filter at analysis instead, a strict superset that reproduces the preregistered metric while enabling the sensitivity checks from one polite pass. That deviation is load-bearing rather than benign: recording every rule is the only reason S2 was computable at all, so anyone rebuilding this scan with axe's runOnly filter set to the WCAG tags could not reproduce the falsification we report above. The two-pass re-scan departs from our preregistered "no retries" rule. We attempted 396 domains rather than 400 because 4 never resolved.
• Machine detectable only. axe-core covers a minority of WCAG success criteria. A clean scan is not an accessible site.
• Homepages only, one page per domain. Interior pages, forms, checkout flows and authenticated areas are unmeasured. Homepages are not whole sites, and the barriers that matter most are often deeper in a flow.
• Single point in time, 2026-07-20, single Tranco daily list. No trend claim is supported.
• Engine specific. Results reflect axe-core 4.12.1 rules and conventions. A different engine yields different numbers, and the WebAIM comparison is cross-engine.
• Small band. The 1 to 100k band is n = 29. Band comparisons are directional at best.
• Aggregates only. We report no individual domain and identify no site as failing. No client or competitor sites were scanned.

What we would actually do with this

The defensible reading is narrower than the headline anyone would want to write from it. Prevalence is trustworthy: roughly nine in ten homepages fail an automated check, and that survives every convention we tested. Composition is not. Fixing contrast, link names and image alternatives will remove most error instances. It will not remove most distinct defect types, and it will not touch the criteria a machine cannot evaluate at all.

So the model that follows is layered, not automated. An automated gate on every deploy catches the high volume mechanical failures before they ship, which is most of the error instances and the cheapest wins available. Human and assistive technology testing judges what the scanner is structurally incapable of judging, which is most of the rulebook. Neither layer substitutes for the other.

That first layer is what we built Vigil to be: 18 automated WCAG 2.2 AA checks that run pre-deploy, so non-compliant changes cannot ship. Given everything above, we want to be precise about what that means. It is a gate against the failure categories machines detect reliably. It is not compliance, and no automated tool is. Our own accessibility conformance statement sets out where we stand.

If you want to talk through where an automated gate fits against the parts of WCAG that still need a human, Contact us and we will give you a straight answer about both.

Citations

1. Tranco "Tranco top-1M list 64YYX" (list created 2026-07-19)
2. WebAIM (Institute for Disability Research, Policy and Practice, Utah State University) "The WebAIM Million: The 2026 report on the accessibility of the top 1,000,000 home pages" (updated 2026-03-30)
3. Fischer, Lundell & Gamalielsson, Universal Access in the Information Society 24(4) "Coverage of web accessibility guidelines provided by automated checking tools" (2025-09-18)
4. W3C Web Accessibility Initiative (WAI) "Selecting Web Accessibility Evaluation Tools" (updated 2024-05-13)
5. U.S. Federal Trade Commission "FTC Order Requires Online Marketer to Pay $1 Million for Deceptive Claims that its AI Product Could Make Websites Compliant with Accessibility Guidelines" (2025-01-03)
6. U.S. Federal Trade Commission "FTC Approves Final Order Requiring accessiBe to pay $1 Million" (2025-04-22)
7. Seyfarth Shaw LLP (ADA Title III blog) "Federal Court Website Accessibility Lawsuit Filings Bounce Back in 2025" (2026-03-25)
8. U.S. Department of Justice, Federal Register 91 FR 20902 "Extension of Compliance Dates for Nondiscrimination on the Basis of Disability; Accessibility of Web Information and Services of State and Local Government Entities" (2026-04-20)