Test Strategy and the Test Pyramid

pytest mechanics (fixtures, parametrize, mocking — see ../pytest/) are how you write tests. Strategy is what to test, at what level, and why — the senior question.

The test pyramid

A model for the shape of a test suite — many cheap fast tests at the bottom, few expensive slow tests at the top:

        /\        E2E         few — slow, brittle, high-fidelity
       /  \
      /----\      Integration  some — real DB/services, moderate speed
     /      \
    /--------\    Unit          many — fast, isolated, cheap

The pyramid’s point: get fast feedback from many unit tests, confidence from fewer integration tests, and only a thin layer of E2E because they’re slow and flaky.

The pyramid vs the “testing trophy”

A common modern critique (Kent C. Dodds’ “testing trophy”): for typical backend services, integration tests are the highest-value tier — pure unit tests over heavily-mocked code can pass while the system is broken (the mocks lied). Many teams now write fewer pure unit tests and more integration tests, especially with fast test databases.

The senior framing: it’s not pyramid-vs-trophy dogma — it’s “push tests as low as they can go while still catching real bugs.” A pure function: unit test it. A query: you must hit a real database or you’re testing your mock of the database. An auth flow across middleware + token validation + a DB lookup: integration. Match the level to where the risk actually is.

Test against behavior, not implementation

The single most important strategy principle.

• Behavior — what the code does, observed through its public interface. “POST /orders with valid input creates an order and returns 201.”
• Implementation — how it does it. “the handler calls _validate() then _persist() then _notify().”

Tests coupled to implementation break on every refactor even when behavior is unchanged — which makes refactoring scary and makes the test suite a liability instead of a safety net. A refactor (by definition behavior-preserving — see ../../26_code_quality/05_refactoring.md) should leave behavior tests green.

Symptoms of implementation-coupled tests: assertions on private method calls, mocks asserting internal call sequences, tests that break when you rename or extract a function.

What to test — risk-driven, not coverage-driven

You can’t test everything; test what matters:

• Core business logic — the rules that, if wrong, cost money or trust. Test thoroughly, including edge cases.
• Boundaries — where untrusted data enters (API input, deserialized messages, config). Test the validation and the rejection paths.
• Error paths — the DB times out, the input is malformed, the dependency 500s. These are less tested than happy paths and break more in prod.
• Things that broke before — every production incident should leave behind a regression test.
• Concurrency / idempotency — if a handler must be idempotent, test that running it twice is safe.

What not to over-test: trivial getters, framework code (you’re not testing Django’s ORM), generated code, glue with no logic.

Coverage — a signal, not a target

Line coverage tells you what code ran during tests — not that it’s correctly tested. You can have 100% coverage with zero assertions.

• Use coverage to find untested code (“the entire error-handling branch has 0% coverage — concerning”).
• Don’t make a coverage number the goal — chasing 100% produces low-value tests for trivial code and trains people to write assertion-free tests that just execute lines.
• Diff coverage (diff-cover) is more useful than total coverage — “is the code this PR changed tested?” — see 04_test_data_and_ci_parallelization.md.
• Branch coverage > line coverage — it checks both sides of every if.

A reasonable posture: track coverage, gate on diff coverage (new code must be tested), don’t gate on a total-coverage number.

The cost side of tests

Tests are an asset and a liability — they cost time to run and to maintain. A good suite is:

• Fast — slow suites get run less, skipped, or -x’d out. Unit tests in milliseconds; the full suite in minutes, not hours (parallelize — see file 04).
• Reliable — a flaky test that fails 1% of the time trains the team to ignore failures, which destroys the suite’s value. Flaky tests are bugs; fix or quarantine them.
• Maintainable — behavior-focused, not implementation-coupled; clear names; minimal setup duplication (fixtures, factories).
• Deterministic — no dependence on wall-clock time, network, ordering, or shared mutable state between tests.

A practical strategy

1. Unit-test the logic that’s genuinely unit-testable (pure-ish functions, business rules) — fast, many.
2. Integration-test the risky wiring — anything touching the DB, the cache, the queue, auth — against real (containerized) dependencies, not mocks. This is where most of the confidence comes from in a backend service.
3. A thin E2E layer over the 3-5 most critical user flows (signup, checkout, the core action). Slow and a bit flaky — keep it minimal.
4. Every incident → a regression test.
5. Gate CI on the suite passing + diff coverage; don’t gate on a coverage percentage.

Common gotchas

• Inverted pyramid — mostly slow E2E tests, few unit tests. Slow feedback, flaky CI, hard to localize failures.
• All unit tests, everything mocked — the suite is green and the system is broken because the mocks didn’t match reality. Push tests down only as far as they still catch real bugs.
• Implementation-coupled tests — break on every refactor; turns the safety net into a refactoring tax.
• Coverage as a target — produces assertion-free line-execution tests; chase diff coverage instead.
• Flaky tests tolerated — trains the team to ignore red CI; a flaky test is a bug.
• Slow suite — gets skipped. Speed is a feature of a test suite.
• No regression test after an incident — the same bug ships again.

Interview angle

• “What’s the test pyramid?” — a model for suite shape: many fast cheap unit tests, fewer integration tests, a thin layer of slow E2E tests. The goal is fast feedback from the bottom and confidence from the middle, with E2E kept minimal because it’s slow and flaky.
• “Pyramid or testing trophy?” — not dogma either way: push each test as low as it can go while still catching a real bug. For backend services, integration tests are often the highest-value tier — pure unit tests over heavy mocks can pass while the system is broken. A pure function → unit; a query → must hit a real DB; an auth flow → integration.
• “Test behavior or implementation?” — behavior, through the public interface. Implementation-coupled tests break on every refactor even when behavior is unchanged, which turns the suite from a safety net into a refactoring tax. A behavior-preserving refactor should leave behavior tests green.
• “How do you decide what to test?” — risk-driven: core business logic, boundaries where untrusted data enters, error paths (under-tested, break more in prod), things that broke before (every incident → a regression test), and idempotency where required. Skip trivial getters and framework code.
• “Is high test coverage good?” — coverage tells you code ran, not that it’s correctly tested — you can have 100% coverage with no assertions. Use it to find untested code; gate on diff coverage (new code is tested) rather than a total-coverage number, which just produces low-value tests.
• “What makes a test suite a liability instead of an asset?” — slow (gets skipped), flaky (trains the team to ignore red), implementation-coupled (breaks on every refactor), or non-deterministic. A good suite is fast, reliable, behavior-focused, and deterministic.