backend / testing / strategy / 04_test_data_and_ci_parallelization.md

Test Data, Flaky Tests, and CI Parallelization

6 interview angles 6 min read source

Test Data, Flaky Tests, and CI Parallelization

The operational side of a test suite: building test data without brittle fixtures, killing flaky tests, and keeping the suite fast as it grows.

Test data — factories over fixtures

A test needs a “user,” an “order,” a “subscription.” Two ways to get one:

  • Static fixtures (a JSON file, a hardcoded dict) — break when the model changes, drift from reality, and force every test to care about every field even when it only cares about one.
  • Factories — code that builds a valid object with sensible defaults, overriding only what the test cares about.
# factory_boy
import factory
from myapp.models import User, Order

class UserFactory(factory.Factory):
    class Meta:
        model = User
    name  = factory.Faker("name")
    email = factory.Faker("email")
    tier  = "free"

class OrderFactory(factory.Factory):
    class Meta:
        model = Order
    user   = factory.SubFactory(UserFactory)   # builds a valid related User
    amount = 100
    status = "pending"

# in a test — override ONLY what this test is about:
def test_high_value_order_flags_review():
    order = OrderFactory(amount=100_000)        # everything else is a sensible default
    assert needs_manual_review(order) is True

Why factories win:

  • The test states only what’s relevant. OrderFactory(amount=100_000) — the reader instantly sees this test is about a high amount; the user, status, etc. are noise the factory handles.
  • Resilient to model changes — add a required field to Order, update the factory once, every test keeps working.
  • SubFactory builds valid object graphs (an order needs a user needs an account…) without each test wiring it by hand.
  • Faker generates realistic varied data instead of "test" everywhere.

Tools: factory_boy (the standard, integrates with Django/SQLAlchemy), model-bakery (Django), or hand-rolled builder functions for simple cases.

Keep factory data deterministic where it matters — if a test asserts on a value, set it explicitly; let Faker randomize only the fields the test doesn’t care about. (And see property-based testing in ../pytest/ for the case where you want the randomization to be the test.)

Flaky tests — a flaky test is a bug

A flaky test passes and fails non-deterministically on the same code. It’s not a minor annoyance — it’s corrosive: it trains the team to re-run CI until green, which means a real failure also gets re-run-until-green and shipped. One tolerated flaky test degrades the trustworthiness of the entire suite.

Common causes:

  • Time — datetime.now(), sleeps, timing assumptions. Freeze time (freezegun, time-machine); never assert on real durations.
  • Order dependence — test B passes only if test A ran first (shared state, leftover DB rows). Tests must be independent and order-shuffleable (pytest-randomly surfaces this).
  • Shared mutable state — module-level globals, a shared cache, a shared DB row not cleaned up. Isolate (transaction rollback per test — see 02_integration_and_contract_testing.md).
  • Concurrency / races — async tests, threads; non-deterministic interleaving.
  • External dependencies — a real network call, a real third-party API. Mock or record them.
  • Resource leaks across tests — an unclosed connection, a port not released.

Handling them:

  1. Fix it — find the root cause (the causes above are a checklist). This is the right answer.
  2. Quarantine, don’t ignore — if you can’t fix it now, mark it (@pytest.mark.flaky / a quarantine list) so it’s tracked and excluded from the gate, not silently re-run. A tracked quarantine is honest; a silent retry is rot.
  3. Re-run as a diagnostic, not a fix — pytest-rerunfailures can confirm flakiness, but auto-retry as a permanent policy just hides the bug.

The cultural point for a senior: don’t tolerate a known flaky test in the gating suite. Fix or quarantine — never “just re-run.”

Keeping the suite fast — CI parallelization

A suite that takes 40 minutes gets run less, skipped, and -x’d. Speed is a feature.

  • pytest-xdist — runs tests across multiple processes/cores: pytest -n auto. The single biggest local + CI speedup. Requires tests to be independent (which they should be anyway) — pytest-xdist distributes them, so order-dependent tests break loudly under it (a feature — it surfaces the coupling).
  • Test database per worker — parallel workers each need their own DB (or schema) so they don’t collide; testcontainers + per-worker DB, or a template database cloned per worker.
  • Split the suite by tier — fast unit tests run on every push; slow integration/E2E run in a separate job, or on a merge queue, or nightly. Don’t make every developer wait on the E2E suite for a one-line change.
  • Test sharding in CI — split the suite across N CI runners (pytest --splits N --group K, or CI-native sharding), each runs 1/N of the tests, total wall-clock drops ~N×.
  • Cache aggressively — the dependency install layer, the mypy/pytest caches, the Docker base image. A CI run shouldn’t reinstall the world every time.
  • Fail fast for the inner loop, run-all for the gate — pytest -x (stop on first failure) for quick local feedback; the full parallel run for the actual CI gate.

Diff coverage in CI

Total coverage as a gate produces low-value tests (see 01_test_pyramid_and_strategy.md). Diff coverage is the better gate:

# diff-cover: was the code THIS PR changed actually tested?
diff-cover coverage.xml --compare-branch=main --fail-under=80

It asks “is the new/changed code tested?” — which is the question that matters per-PR — instead of holding a PR hostage to the whole codebase’s historical coverage number.

Test selection — run only what’s affected

For large suites, running everything on every change is wasteful. Affected-test selection runs only tests touching the changed code:

  • pytest-testmon — tracks which tests cover which code, runs only the affected ones on a change. Great for the local inner loop.
  • CI usually still runs the full suite on the gate (correctness over speed for the merge), but the inner-loop testmon run keeps developers fast.

Common gotchas

  • Static fixtures everywhere — brittle, drift from the model, force tests to care about irrelevant fields. Factories with sensible defaults + targeted overrides.
  • Tolerating a flaky test in the gate — trains the team to re-run-until-green, so real failures ship. Fix it, or quarantine it visibly — never silently retry.
  • Order-dependent tests — test B needs test A’s leftovers. They must be independent; pytest-randomly and pytest-xdist surface the coupling.
  • Time-dependent tests — datetime.now(), sleeps, duration assertions. Freeze time; never assert on wall-clock.
  • A 40-minute suite — gets skipped. pytest-xdist, CI sharding, tier splitting, caching.
  • Total-coverage gate — produces assertion-free tests; gate on diff coverage.
  • Parallel workers sharing one test DB — collisions, flakiness. DB-per-worker.

Interview angle

  • “Factories or fixtures for test data?” — factories (factory_boy): a test builds an object with sensible defaults and overrides only the field it’s about, so the test states only what’s relevant and survives model changes. Static fixtures are brittle, drift from the model, and force every test to care about every field.
  • “A test fails intermittently — what do you do?” — treat it as a bug, because a tolerated flaky test trains the team to re-run-until-green and that ships real failures. Find the root cause (time, order dependence, shared state, races, external calls) and fix it; if you can’t immediately, visibly quarantine it — never silently auto-retry.
  • “Common causes of flaky tests?” — datetime.now()/sleeps (freeze time), order dependence and shared mutable state (isolate per test, randomize order to surface it), concurrency races, real external calls (mock/record), and resource leaks across tests.
  • “Your test suite takes 40 minutes. What do you do?” — pytest-xdist to parallelize across cores, CI sharding across runners, split fast unit tests (every push) from slow integration/E2E (separate job/nightly), cache the dependency-install and test caches, and give each parallel worker its own test DB.
  • “How do you gate coverage in CI?” — on diff coverage (diff-cover) — “is the code this PR changed tested?” — not total coverage, which just produces assertion-free line-execution tests and holds PRs hostage to historical numbers.
  • “How do you keep the local inner loop fast?” — pytest -x to fail fast, pytest-testmon to run only tests affected by the change, and pytest-xdist for parallelism. CI still runs the full suite on the gate — speed for the developer, completeness for the merge.
Previous
Load and Performance Testing