A11y Testing + Visual Regression in CI

TL;DR

Two CI gates that prevent classes of regression code-review can’t catch:

• A11y testing with axe-core — runs static analysis on rendered components (in unit tests) or full pages (in E2E) and flags violations: missing <label>, low contrast, invalid ARIA, focus traps, etc. Catches roughly 30-50% of WCAG issues automatically; the rest needs manual review.
• Visual regression — captures screenshots and diffs against a baseline. Detects pixel-level changes (color, layout, font swap) that behavior tests miss.

Both are deltas, not absolute checks — they flag changes that need review. Used together with role-based RTL queries and Playwright E2E, they form a layered defense against the bug classes pure unit tests can’t see.

A11y Q&A

Q: How does axe-core work?

A: A standalone JS engine that walks the DOM and checks against WCAG rules. Returns a list of violations with element selectors, rule IDs, severity, and remediation hints.

import { axe } from "vitest-axe";
import { render } from "@testing-library/react";

it("login form is accessible", async () => {
  const { container } = render(<LoginForm />);
  const results = await axe(container);
  expect(results).toHaveNoViolations();
});

The matcher prints a readable list of violations on failure:

Expected the HTML to have no violations:
- color-contrast: Elements must have sufficient color contrast (selector: button.btn)
- label: Form elements must have labels (selector: input#search)

Q: What does axe catch and what does it miss?

A:

Catches (~30-50% of WCAG issues):

• Missing <label> on form inputs.
• Insufficient color contrast (text vs background).
• Missing alt on <img>.
• Invalid ARIA roles / attributes.
• Heading order violations (<h1> → <h3> skipping <h2>).
• Duplicate id attributes.
• Form controls not associated with labels.

Misses (need manual review):

• “This component is keyboard-navigable” — axe doesn’t tab through.
• “The screen-reader announcement is meaningful” — axe can’t read context.
• “Focus management on modal open/close is correct.”
• “Dynamic content is announced via aria-live.”
• Whether the layout makes sense visually for users with low vision.

The senior framing: axe is the floor, not the ceiling. Pair with periodic manual screen-reader testing (VoiceOver on macOS, NVDA on Windows) and keyboard-only navigation runs.

Q: Per-component vs per-page a11y testing.

A:

Per-component (in unit tests) — catches issues early, scoped to the component:

it("button is accessible", async () => {
  const { container } = render(<Button>Save</Button>);
  expect(await axe(container)).toHaveNoViolations();
});

Per-page (in E2E) — catches integration issues (z-index conflicts, focus traps, real CSS contexts):

import { test, expect } from "@playwright/test";
import AxeBuilder from "@axe-core/playwright";

test("homepage has no a11y violations", async ({ page }) => {
  await page.goto("/");
  const results = await new AxeBuilder({ page }).analyze();
  expect(results.violations).toEqual([]);
});

Both have a place. Per-component catches issues at the unit; per-page catches integration bugs.

Q: Storybook + a11y addon.

A: @storybook/addon-a11y runs axe against each story in the Storybook UI; violations appear in an “Accessibility” panel.

// stories/Button.stories.ts
export const Primary: Story = {
  args: { children: "Save" },
  parameters: {
    a11y: {
      config: {
        rules: [{ id: "color-contrast", enabled: true }],
      },
    },
  },
};

For CI: @storybook/test-runner + axe runs every story through axe automatically. Catches a11y regressions per-state (default, hover, disabled, error).

Q: Specific axe rules to know.

A: A senior should be able to name the common ones:

Q: When to disable an axe rule.

A: Only with documented reason. The disable should be at the most specific level (single test, single component) and reviewed periodically:

const results = await axe(container, {
  rules: {
    "color-contrast": { enabled: false },         // brand colors approved separately
  },
});

Blanket disables (project-wide) are a smell — usually means the team gave up. Either fix or accept and document.

Visual Regression Q&A

Q: What is visual regression testing?

A: Capture a screenshot of a component/page. Save as “baseline.” On subsequent runs, capture again, pixel-diff against the baseline. Flag any differences for human review.

// Playwright
await expect(page).toHaveScreenshot("homepage.png");

First run: creates baseline. Subsequent runs: compare, fail on diff (with a threshold).

Q: Tooling options.

A:

For Storybook-centric teams: Chromatic is the smoothest UX. For Playwright E2E coverage: built-in screenshots are free. Mix as needed.

Q: What catches visual regression that other tests miss?

A:

• CSS regressions — refactored Tailwind utility caused a color change.
• Font swap layout shift — font loaded with different metrics.
• Layout breaks at viewport — flex/grid bug at mobile/tablet.
• Z-index bugs — modal disappearing behind a sticky header.
• Theme regressions — dark mode broken on one component.
• Browser-specific rendering — Safari blurring a transform.

Behavior tests pass; visual regression catches. Worth it especially when shipping a design system to many consumers.

Q: Per-state visual regression with Storybook.

A: Each Storybook story is a screenshot target. Chromatic / Loki / @storybook/test-runner snap every story:

// Button.stories.ts
export const Default: Story = { args: { children: "Save" } };
export const Disabled: Story = { args: { children: "Save", disabled: true } };
export const Loading: Story = { args: { children: "Save", loading: true } };
export const Long: Story = { args: { children: "Save a really long button label" } };

Every state visually verified per PR. Refactors that affect appearance fail loudly.

Q: Multi-viewport visual regression.

A: Playwright lets you screenshot at multiple viewports per test:

test.describe("homepage", () => {
  for (const viewport of [
    { width: 375, height: 667, name: "mobile" },
    { width: 768, height: 1024, name: "tablet" },
    { width: 1440, height: 900, name: "desktop" },
  ]) {
    test(`renders ${viewport.name}`, async ({ page }) => {
      await page.setViewportSize(viewport);
      await page.goto("/");
      await expect(page).toHaveScreenshot(`home-${viewport.name}.png`);
    });
  }
});

Catches responsive bugs that a single-viewport test misses.

Q: Handling dynamic content in screenshots.

A:

• Mask animations / dates / random IDs with mask: [locator] in Playwright.
• Mock time (vi.setSystemTime) before rendering so dates are stable.
• Disable animations in the test config:

  await page.addStyleTag({ content: `*, ::before, ::after { animation: none !important; transition: none !important; }` });

• Wait for fonts to load before screenshot (document.fonts.ready).

Without these, every run produces a slightly different screenshot and you live in --update-snapshots hell.

Q: How do you review visual diffs?

A:

• Chromatic / Percy: side-by-side, baseline + new + diff, approve/reject per change.
• Playwright: diff image generated as artifact; review in PR. Less ergonomic but free.
• Storybook test-runner: similar to Playwright artifacts.

Don’t review by glancing at “diff %” — actually look at the image. A 2% diff that moves a price by 10 pixels is a real bug.

Gotchas / edge cases

• Flaky screenshots — fonts loading, animations, timing. Disable animations + wait for fonts.
• OS / browser font rendering differs — visual regression often pinned to a single OS in CI (Docker container).
• axe violations on third-party components — file with the vendor; can’t always fix yourself. Document the exception.
• Chromatic’s “TurboSnap” mode only screenshots components affected by a PR — major speedup for large libraries.
• A11y testing skips disabled/hidden content — axe may underreport. Render the “active” state in the test.
• Manual a11y testing isn’t replaced — keyboard nav, screen reader announcements, color blindness simulation. Quarterly audits.

What a senior is expected to say

• “axe-core catches ~30-50% of WCAG issues automatically. It’s the floor, not the ceiling — pair with manual keyboard + screen-reader testing.”
• “Per-component a11y in unit tests; per-page a11y in E2E. Plus Storybook addon for per-state coverage.”
• “Visual regression catches what behavior tests can’t — CSS regressions, layout breaks at viewport, theme bugs, browser-specific rendering.”
• “Disable animations + mock time + wait for fonts in visual tests. Without these, every run is flaky.”
• “Chromatic for Storybook-heavy teams, Playwright built-in for E2E coverage. Pick based on where the components live.”
• “An axe rule disable is documented; project-wide disables are a smell.”

Cross-references

• Accessibility deep dive: ../16_accessibility/
• Playwright E2E: 09_playwright_e2e.md
• RTL role-based queries (the unit-test a11y check by passing): 03_react_testing_library.md
• Storybook (component library docs): ../14_frontend_system_design/09_design_a_component_library.md

Further reading

• axe-core rules: https://github.com/dequelabs/axe-core/blob/develop/doc/rule-descriptions.md
• vitest-axe: https://github.com/chaance/vitest-axe
• @axe-core/playwright: https://playwright.dev/docs/accessibility-testing
• Chromatic: https://www.chromatic.com/
• WCAG 2.2 guidelines: https://www.w3.org/WAI/standards-guidelines/wcag/