Load and Performance Testing

Functional tests answer “is it correct?” Load tests answer “does it stay correct and fast under traffic?” — and “where does it break?” A distinct discipline with its own tools and its own failure modes.

The types of load test — they ask different questions

These are different tests. “Load testing” colloquially means the first, but a senior answer distinguishes them — a system can pass a load test and fail a soak test (slow memory leak) or a spike test (autoscaling too slow).

The tools

For a Python backend team, Locust is the easy pick — the load scenario is Python, so it lives next to the code and the team can read it:

from locust import HttpUser, task, between

class APIUser(HttpUser):
    wait_time = between(1, 3)          # think time between requests

    @task(3)                           # weight 3 — 3× as common as the task below
    def view_feed(self):
        self.client.get("/feed", name="/feed")

    @task(1)
    def create_post(self):
        self.client.post("/posts", json={"body": "hello"}, name="/posts")

Designing a load test that means something

A load test that doesn’t model reality produces numbers you can’t trust:

• Realistic traffic mix — weight the scenarios like real users (mostly reads, occasional writes), not a uniform hammer on one endpoint.
• Think time — real users pause between actions (wait_time). Zero think time tests an unrealistic pattern and inflates the apparent load per user.
• Realistic data — vary the inputs (different user ids, different payloads). Hammering one cached key tells you about the cache, not the system. Cold-cache vs warm-cache behavior differs hugely.
• Realistic ramp — ramp users up gradually; a step to full load tests a different thing (a spike) than steady-state.
• Test environment ≈ prod — load-testing a 1-CPU dev box tells you nothing about prod. Use a prod-like environment, or accept the numbers are only directional.
• Don’t load-test prod carelessly — if you must, use a small percentage of synthetic traffic, off-peak, with a kill switch.

What to measure — and the p99 point

Always look at percentiles, not the average. A 20ms average can hide a 2-second p99 — and at scale the p99 is millions of real requests. The pass/fail criterion is a percentile target (“p99 < 200ms at 5k RPS”), never an average. See ../../28_networking/ and the system-design latency file.

Finding the bottleneck — load test + observability together

A load test that just says “it fell over at 3k RPS” is half the value. The other half is why. Run the load test with the system’s observability on (metrics, traces, profiler) and watch what saturates first:

• CPU pegged → compute-bound; profile the hot path, or scale out.
• DB connections exhausted → the connection-pool / Lambda-connection-storm problem; pool tuning, RDS Proxy.
• Memory climbing and not falling → a leak (this is what a soak test catches that a short load test doesn’t).
• Queue depth growing unbounded → consumers can’t keep up; scale workers.
• Latency rises but CPU is low → you’re waiting on something — a lock, a slow downstream, a thread-pool starved by sync calls in async code.
• A downstream service’s latency climbs → the bottleneck moved; it’s not in your service.

The senior skill is reading “the load test failed” as “now find which resource saturated” — the test locates the bottleneck; it doesn’t just grade pass/fail.

Soak tests catch what load tests miss

A 5-minute load test passes; the service still falls over after 6 hours in prod. That’s a soak test finding — slow degradation that only shows over time:

• Memory leaks — slowly climbing RSS until OOM.
• Connection / file-descriptor leaks — pool slowly exhausts; works fine until it doesn’t.
• Unbounded growth — a cache with no eviction, a table that’s only ever inserted into, log disk filling.
• Resource fragmentation — performance slowly decaying.

Run a soak test (hours, steady moderate load) before trusting a service in production. It’s the test most teams skip and most regret skipping.

Load testing in CI

• A full load test is too slow/expensive for every PR. Run it nightly, or pre-release, against a staging environment.
• A lightweight smoke load test (a short k6/Locust run with a modest latency assertion) can run per-PR — it catches a gross performance regression (“this PR made the endpoint 10× slower”) even if it can’t characterize full capacity.
• Set a regression budget — fail the build if p99 regresses beyond a threshold versus the baseline. Performance regressions caught in CI are cheap; caught in prod are an incident.

Common gotchas

• Averaging instead of percentiles — a fine average hides a terrible p99, which is the real user experience at scale.
• Unrealistic traffic — uniform hammering of one endpoint, no think time, one cached key. The numbers don’t transfer to prod.
• Load-testing a non-prod-like environment — a 1-CPU dev box result tells you nothing about prod capacity.
• No observability during the test — you learn that it broke, not why. Run the test with metrics/traces/profiler on.
• Skipping the soak test — short load tests miss leaks; the service passes CI and dies after hours in prod.
• Ignoring error rate — “10k RPS!” while half the responses are 500s is not 10k RPS of service.
• No CI regression gate — a perf regression ships silently and becomes a prod incident.

Interview angle

• “What types of load testing are there?” — load (meets targets at expected peak?), stress (where and how does it break?), spike (sudden traffic jump?), soak/endurance (degrades over hours? leaks?), scalability (does adding capacity actually help?). They ask different questions — a system can pass a load test and fail a soak test.
• “What do you measure in a load test?” — throughput (sustained RPS), latency percentiles (p50/p95/p99/p99.9 — never the average, which hides the tail), error rate (fast-because-it’s-500ing isn’t fast), and resource saturation (CPU, DB connections, memory, queue depth) to locate the bottleneck.
• “Why percentiles instead of the average?” — the average hides the tail; a 20ms average can mask a 2s p99, and at scale the p99 is millions of real users. The pass/fail criterion is a percentile target, e.g. “p99 < 200ms at 5k RPS.”
• “Your load test shows latency rising but CPU is low — what does that mean?” — you’re waiting, not computing: a lock, an exhausted connection pool, a slow downstream, or sync calls starving an async thread pool. Run the load test with observability on so the test locates the saturated resource, not just grades pass/fail.
• “What does a soak test catch that a load test doesn’t?” — slow degradation over time: memory leaks, connection/FD leaks, unbounded cache/table growth, disk filling. A 5-minute load test passes; the service still dies after 6 hours. It’s the test teams skip and regret.
• “How do you load-test in CI?” — full load tests are too slow per-PR; run them nightly/pre-release against staging. A lightweight smoke load test per PR with a p99 regression budget catches gross regressions — a perf regression caught in CI is cheap, caught in prod is an incident.
• “How do you design a load test that’s actually meaningful?” — model reality: realistic traffic mix (weighted reads/writes), think time between requests, varied inputs (not one hot cached key), gradual ramp, and a prod-like environment. An unrealistic test produces numbers that don’t transfer.