Test automation frameworks and API testing

The engineering half of an automation QA role: how you build a suite that stays useful, and how you test an API properly.

Framework architecture

A “framework” here means the structure around the tests, not the runner. Layers, from the bottom:

runner            pytest
fixtures/setup    environment config, auth, test data lifecycle
clients           typed API client / page objects wrapping the system under test
test data         builders and factories, not shared fixtures files
tests             readable scenarios, one behaviour each
reporting         JUnit XML / Allure, published from CI

The rule that keeps a suite maintainable: tests express intent, layers below express mechanics. A test that contains a URL, a JSON payload and a status code assertion breaks on every unrelated API change. A test that calls orders.create(customer=c, items=[i]) does not.

For UI, the Page Object Model is the same idea — selectors and interactions live in a page class, tests describe the journey. Its successor in Playwright-style suites is role-based locators plus small helper objects, which achieve the same decoupling with less ceremony.

Choosing the runner

pytest is the right default for a Python automation stack: fixtures give you scoped setup and teardown, parametrize gives data-driven tests, and the plugin ecosystem covers parallelism (pytest-xdist), retries, HTML reports and Allure integration. See ../pytest/.

@pytest.mark.parametrize("qty,expected", [(1, 200), (0, 422), (10_001, 422)])
def test_order_quantity_validation(api, qty, expected):
    r = api.post("/orders", json={"sku": "ABC", "qty": qty})
    assert r.status_code == expected

unittest when you must match an existing codebase; behave/pytest-bdd when non-technical stakeholders genuinely read the scenarios — and not otherwise, because Gherkin adds a translation layer that costs more than it returns if only engineers ever look at it.

API testing

What to assert, beyond a 200:

That last row is the one worth volunteering. Broken object-level authorisation is the top item in the OWASP API Security Top 10, and testing it means calling endpoints as user A with user B’s resource ids — something a purely functional suite never does.

Contract testing is the layer above. Consumer-driven contracts (Pact) or schema-diffing an OpenAPI spec in CI catches provider changes that break consumers, without a full integration environment. For internal microservices this is usually higher value than more end-to-end tests. See ../strategy/02_integration_and_contract_testing.md.

Tools: httpx or requests in pytest for the suite itself; Postman/Newman where a shared collection is the team’s working artefact; Schemathesis to generate property-based tests directly from an OpenAPI spec, which finds edge cases nobody wrote by hand.

Test data

The most common cause of a suite that works locally and fails in CI.

• Create what you need, in the test, then clean up. A test depending on a record someone loaded last month will fail eventually.
• Builders with sensible defaults so a test overrides only the field it cares about: UserFactory(role="admin").
• Isolation — a per-test transaction rolled back, a per-worker schema, or a container per run. Shared mutable state is what makes tests order-dependent.
• Never point automation at production data. Beyond the obvious risk, in a regulated context it is a privacy incident. See ../../31_healthcare_regulated/02_phi_privacy_and_secure_coding.md.

testcontainers is the strongest general answer for integration dependencies: a real Postgres, Redis or Kafka per run, disposable and identical in CI and locally.

UI and cross-browser

Playwright is the current default: auto-waiting (which removes the largest source of flakiness), Chromium/Firefox/WebKit from one API, tracing and video for failure diagnosis, and parallel execution by default. Selenium remains where an existing suite or a specific grid infrastructure requires it.

For real device and browser coverage, a cloud grid (BrowserStack, Sauce Labs, LambdaTest) beats maintaining your own. Keep the UI layer thin: a small number of critical journeys end to end, with everything else covered at the API layer, which is faster and far less brittle. See ../../../frontend/10_testing/09_playwright_e2e.md.

Visual regression (Playwright snapshots, Percy, Applitools) catches what functional assertions cannot — but needs a tolerance policy and a review workflow, or every font-rendering difference becomes a false failure.

Performance, load, and stress

Different questions, and interviewers check you distinguish them:

k6 (JS scripting, good CI integration) and Locust (Python, so the team already knows the language) are the usual choices; JMeter where it is already established.

Measure percentiles, not averages — p95 and p99 are what users experience, and an average hides the tail entirely. Define the SLO before the test, or you are just generating numbers. See ../strategy/03_load_and_performance_testing.md.

Running it in CI

• Fast feedback first. Unit and API tests on every push; the full UI suite on merge or nightly.
• Parallelise with pytest-xdist or the CI’s own sharding, which requires test independence — a good forcing function.
• Publish artefacts on failure: traces, screenshots, videos, logs. A CI failure you cannot diagnose without re-running locally wastes the whole point.
• Fail the build on real failures, quarantine flakes explicitly rather than adding blanket retries. Blanket retry-on-failure hides genuine intermittent bugs.
• Report trends, not just pass/fail: duration, flake rate, coverage of critical paths.

Interview angle

• “How do you structure an automation framework?” - layered: runner, fixtures and config, a client or page-object layer wrapping the system, test-data builders, then tests that read as intent. The point is that an API or UI change touches one layer, not every test.
• “What do you assert on an API response beyond the status code?” - schema conformance, semantic correctness of values, headers, error cases, idempotency, pagination boundaries, and cross-tenant authorisation. That last one is the top OWASP API risk and is the answer that stands out.
• “How do you keep tests from becoming flaky?” - wait on conditions rather than sleeping, isolate test data per test, keep tests order-independent so they can run in parallel, and use a framework with auto-waiting. Then quarantine and fix flakes instead of adding retries.
• “Load, stress or soak?” - load validates SLOs at expected traffic, stress finds the breaking point and failure mode, soak reveals leaks and resource exhaustion over time. Report percentiles, and define the SLO before running anything.
• “How much end-to-end UI testing?” - a small set of critical journeys, with the bulk of coverage at the API layer. UI tests are the slowest and most brittle, so their value has to be earned per test.
• “How do you provision test dependencies?” - testcontainers for a real Postgres, Redis or Kafka per run. Mocks for third parties you do not control, plus contract tests so the mock cannot silently drift from the real provider.