Testing accessibility

Short answer

Automated accessibility tools quickly find problems that can be read in the code and decided by a single rule: the contrast of text over a defined background color, an image with no alt text, a form field with no label. What they usually can't catch is how readable the text over a photo actually is, whether a piece of information is conveyed by color alone, whether an error sentence is understandable, and whether a focus ring is visible to the user's eye; in a 2017 trial by the UK government, none of ten tools found 42 of 143 deliberately planted barriers. That's why color accessibility is tested on three layers: tools, review by a human eye, and tests with real users who have low vision, are older, or see color differently.

What the tool sees, and what it doesn't

An automated accessibility tool reads a page's code and computed styles and tests every element against a list of rules. If a paragraph's text color and background color are defined in the stylesheet, the tool calculates the ratio with UX-06's formula and flags anything that falls below the threshold. This turns hours of human work into seconds, and it can be redone on every build.

The tool's limit follows from this: it sees what can be decided in the code. When text sits over a photo or a gradient, there's no single defined background color in the stylesheet; the tool either says it can't measure, or it looks at a fallback color and gives a wrong result. Knowing whether a piece of information is conveyed by color alone requires knowing what that color means. Whether an error sentence reads "Invalid input" or "Date of birth must be in day/month/year format" is a language decision.

The UK government's digital service team ran an experiment to measure this limit. The team built a web page deliberately filled with accessibility errors and called it "the world's least-accessible webpage." The page had 143 errors across nineteen categories. It was scanned with ten widely used free tools. None of the tools found 42 of the 143 barriers — 29 percent. Among the ones missed were links identified by color alone; the underline problem covered in UX-07 was one of the issues no tool caught. The best-performing tool found 37 percent of the barriers when only errors and warnings were counted, and 41 percent when manual-check prompts were also counted; a popular browser tool found only 17 percent.

This result belongs to the tools of 2017, and tools have improved since then. But the limit the experiment revealed is structural: a significant part of what a tool can't see lies in meaning, not in code. The team's own comparison holds true: a spell-checker finds spelling mistakes, but it can't decide whether a sentence is understandable.

Review by a human eye

The first thing that fills the gap the tool leaves is the team's own eye. Manual review is done with a checklist, and the same questions are asked on every page.

The most productive questions, where color is concerned, are these. Does the page read in grayscale, and does every piece of information given by color have a backup? Has every place where text falls over a photo or a gradient been tested against the worst-case pixel? When navigating by keyboard, is the focus ring visible on every element, on both light and dark surfaces? Are hover, pressed, and disabled states distinguishable from each other? Are the same questions asked again in dark theme and in high-contrast mode?

Vision simulations are a powerful aid to this review. Chrome's developer tools let you view a page with simulations for blurred vision, reduced contrast, protanopia, deuteranopia, tritanopia, and achromatopsia. Its documentation states that protanopia and deuteranopia mix up green, red, and yellow, and tritanopia mixes up green and blue.

It's also important to know the simulation's limit. A simulation is the output of a model; it shows the extreme case where the deficiency is fully present. As BI-12 explains, most color vision deficiencies are partial and vary from person to person. A screen that reads in the simulation means a person with a partial deficiency will read it too; but the simulation doesn't show how that person actually experiences the screen, where they struggle, and what workarounds they've developed on their own.

Colors the user chooses

Some users never see the designer's colors at all. Operating systems' high-contrast themes and the user's own chosen color schemes take the place of the page's colors.

The CSS color adjustment specification defines this mode as an accessibility feature designed to increase text readability through color contrast, and it states that many people with low vision find it more comfortable to read when there's a specific kind of contrast between the foreground and the background. The specification notes that users can set up their own themes with not just high contrast but low contrast, or with only a difference in hue.

This rule exposes two traps teams often fall into. Drawing the focus ring with box-shadow is one of them: because shadows are ignored in forced colors mode, the ring disappears entirely, and all of UX-04's care around focus goes to waste. Drawing the ring with the outline property keeps it visible in this mode too. The other trap is giving a button its shape only through a gradient background or a shadow: the gradient is removed, and without a border the button looks like plain text. Giving the button a transparent border ensures that border shows up in the user's chosen color in forced colors mode. A tool usually can't find these traps; the team has to open the page in this mode and navigate it.

Testing with real users

Tools and review ask the questions the team already knows. Testing with real users surfaces the questions the team doesn't know to ask.

Vision problems aren't a small minority's problem. The World Health Organization's 2019 vision report states that at least 2.2 billion people worldwide have a near or distance vision impairment, and that in at least 1 billion of them, the impairment is preventable or has not yet been addressed.

A user test for color accessibility diversifies its participants: people with color vision deficiency, people with low vision who use zoom or high-contrast mode, and older users who use their screen for long stretches and under different lighting. Participants aren't asked "Do you like this color?"; they're given a task, and the team watches where they hesitate: filling out a form and fixing an error, reading a value off a chart, turning a setting on and off. When the test is run on the person's own device with their own settings, a zoom level or color theme the team never tried also enters the process.

The result of these tests is often not a contrast ratio. A participant reading an icon to mean something else, missing a piece of information entirely, or pressing a button repeatedly without realizing it's disabled — these are findings that appear in no tool's report.

Putting testing into the workflow

Each of the three layers runs at a different frequency. Automated audit is the cheapest, and it can run on every code change as a step in the build process; a new low-contrast piece of text or an unlabeled field gets caught before the code is even merged. Manual review is more expensive; it's done with a checklist on every new screen design and before every release. User testing is the most expensive and the most informative; it's planned before a major design change, when a color system or theme is rebuilt, and for the product's most critical flows.

This order isn't an order of importance. Every problem the automated audit finds frees up people's time for harder problems; every problem user testing finds is often added as a new item to the next checklist. Over time, a team starts checking manually, and where possible automatically, for an error it once saw in front of a user.

Example — a headline over a portrait

On a news site's home page, a white headline is written over a portrait photo that changes every day. The team's automated tool scans the page on every build and finds the headline's contrast valid: the stylesheet defines a dark gray, #333333, for the headline's background in case the photo fails to load, and white is comfortably above the threshold against that gray.

But what the headline actually falls over is the photo, and the photo is someone different every day. The math shows a difference: white text gives about 6.1:1 over a cheek in the #8D5524 tone, 12.6:1 over the #4A2C1D tone, but drops to 3.0:1 over the #C68863 tone and 1.6:1 over the #F1C7A9 tone. The same headline reads comfortably over a dark-skinned person's portrait and nearly vanishes over a light-skinned person's; the tool gives the same "passed" report for both days, because it never looks at the photo.

The team fixes the problem in the process, not in the tool. A semi-transparent dark band is added behind the headline so the text clears the threshold independent of the photo. The manual review checklist gets a new item: "if text falls over a photo, measure against the lightest region." For the pre-release check, a test set is prepared covering light, wheat-toned, brown, and very dark skin tones, along with plain and busy backgrounds, and the headline is checked against each one. Finally, the page is opened in forced colors mode: because the band is defined as a solid background color rather than a gradient, it isn't removed in this mode — it turns into the user's chosen background color, and the headline stays readable in the user's text color.

Common misconception

Summary

1. Automated tools find problems that can be decided in the code quickly and repeatably; they usually can't see text over a photo or gradient, information conveyed by color alone, or whether text is understandable.
2. Manual review is done with grayscale, vision simulation, keyboard navigation, and forced colors mode; because shadows and gradients are removed in forced colors mode, the focus ring must be given with outline and a button's border with an actual border.
3. Task-based tests with users who have low vision, are older, or see color differently surface findings no tool reports; text over a photo is tested against the lightest region using a portrait set spanning a wide range of skin tones.

Test yourself

1. (Easy) Of the 143 barriers planted in the UK government's 2017 trial, how many were not found by any tool?
2. (Easy) What options do Chrome DevTools's vision deficiency simulations offer?
3. (Medium) Which accessibility problems can automated tools not catch?
4. (Medium) What happens to a focus ring drawn with box-shadow in forced colors mode, and what should be done instead?
5. (Hard) Why doesn't a vision simulation's result stand in for user testing?
6. (Hard) Why does a headline over a portrait pass the automated tool but go unreadable on some days, and how is the process fixed?

Answers

1. 42, that is, 29 percent ("What the tool sees, and what it doesn't").
2. Blurred vision, reduced contrast, protanopia, deuteranopia, tritanopia, and achromatopsia ("Review by a human eye").
3. They usually can't catch the real contrast of text over a photo or gradient where the stylesheet defines no single background color; whether a piece of information is conveyed by color alone, such as links set apart only by color; whether an error sentence is understandable; and whether a focus ring and state differences are actually visible, or elements that vanish in forced colors mode. These lie in meaning and in the image, not in the code ("What the tool sees, and what it doesn't", "Colors the user chooses").
4. In forced colors mode, box-shadow is ignored and the ring disappears entirely. The ring must be drawn with the outline property; a button's borders must also be given with an actual border rather than a gradient or a shadow ("Colors the user chooses").
5. A simulation is a model's output for the extreme case where a deficiency is fully present; it doesn't show deficiencies that are partial and vary from person to person, a person's own settings and the workarounds they use, or how they interpret an icon. A simulation does the pre-screening; task-based testing surfaces the questions the team doesn't know to ask ("Review by a human eye", "Testing with real users").
6. The tool looks at the #333333 fallback background defined in the stylesheet, but the headline actually falls over the photo. White text gives 12.6:1 over the #4A2C1D tone but drops to 1.6:1 over the #F1C7A9 tone. A solid semi-transparent band is added behind the headline, the review checklist gets a "measure against the lightest region" item, a test set with portraits of different skin tones is prepared, and the page is also opened in forced colors mode ("Example — a headline over a portrait").

Sources

• Mehmet Duran, "What we found when we tested tools on the world's least-accessible webpage," GOV.UK Accessibility blog, February 24, 2017.
• Chrome for Developers, "Apply effects," Emulate vision deficiencies section.
• W3C CSS Working Group, "CSS Color Adjustment Module Level 1," Editor's Draft.
• World Health Organization, "World report on vision," 2019.

Next step

UX-09 moves on to new tools for writing colors: why the steps of a tonal scale built with oklch() look more even than the same steps in HSL, how color-mix() and relative color syntax let a theme be derived from a single value, and how to set up a fallback for browsers that don't support P3 colors by checking support before release. UX-10 then opens the screen onto a game world: how a single yellow bar guides the player across a gray scene, how friend and foe get coded by color, and how games build color-blindness modes.