Timeouts, retries and backoff

The first thing to get right about any network call, and the source of most cascading failures when it’s wrong. The interview question is rarely “what is a retry” — it’s which failures you retry and what stops the retry storm.

Timeouts come first

A retry policy without a timeout is not a resilience strategy. Without a timeout the call hangs, the worker is held, the pool drains, and the failure spreads upstream. That’s how one slow dependency takes down a service that isn’t broken.

Set both halves explicitly:

import httpx

client = httpx.AsyncClient(
    timeout=httpx.Timeout(connect=2.0, read=10.0, write=5.0, pool=1.0),
)

The pool timeout is the one people omit and then can’t explain: when the pool is exhausted, requests queue inside your process, so the caller waits far longer than the read timeout suggests.

Budget downward. If your endpoint has a 3-second SLA and calls three services, each cannot have a 3-second timeout. Allocate a total budget and pass the remaining time down:

async def handler(request):
    async with asyncio.timeout(3.0):          # the whole request budget
        ...

Timeouts that exceed the caller’s own patience are pure waste — you’re computing a result nobody is still waiting for.

Retry only what can succeed

RETRYABLE = (
    httpx.ConnectError, httpx.ConnectTimeout, httpx.ReadTimeout,
    httpx.RemoteProtocolError,
)

def should_retry(exc: Exception, response: httpx.Response | None) -> bool:
    if isinstance(exc, RETRYABLE):
        return True
    if response is None:
        return False
    return response.status_code in {429, 502, 503, 504}

The nuance worth stating: 500 is not automatically retryable. A deterministic server bug returns 500 every time, and if the handler partially applied before failing, retrying duplicates the effect.

The idempotency precondition

Retrying a non-idempotent operation is a correctness bug, not a resilience feature. A retried POST /charges after a timeout can charge twice — and a timeout is exactly the case where you don’t know whether it succeeded.

await client.post(
    "/charges",
    json=payload,
    headers={"Idempotency-Key": str(order.idempotency_key)},   # stable per attempt
)

The key must be stable across retries and derived from the business operation, not generated per attempt. See ../../backend/13_architecture_design/18_idempotency_keys.md.

Safe to retry without a key: GET, PUT with the same body, DELETE. Not safe: POST that creates something, anything that increments a counter or sends a message.

Backoff and jitter

from tenacity import (
    retry, retry_if_exception_type, stop_after_attempt,
    wait_exponential_jitter, before_sleep_log,
)

@retry(
    retry=retry_if_exception_type(RETRYABLE),
    wait=wait_exponential_jitter(initial=0.5, max=10, jitter=2),
    stop=stop_after_attempt(4),
    before_sleep=before_sleep_log(logger, logging.WARNING),
    reraise=True,
)
async def fetch(url: str) -> httpx.Response:
    resp = await client.get(url)
    resp.raise_for_status()
    return resp

Jitter is the part that matters and the part people leave out. Without it, every client that failed at the same moment retries at the same moment: 1s, 2s, 4s — synchronised waves that re-break the service each time it comes up. This is the thundering herd, and it turns a brief blip into a sustained outage.

wait_exponential_jitter randomises the delay. Full jitter (random.uniform(0, computed_delay)) spreads the hardest and is the safest default.

Cap the delay (max=10) so a retry can’t outlive the caller’s budget, and cap attempts (3-4). More attempts against a genuinely down service just extends the failure.

Retry amplification — the thing seniors name

Retries multiply through a call chain. Service A retries 3x into B, B retries 3x into C: C sees 9 requests for one user action. Four layers deep is 81.

Under partial degradation this is how retries cause the outage rather than surviving it — the retry load is what finishes off a service that was merely slow.

Defences:

• Retry at one layer only. Usually the outermost client that owns the SLA, or the innermost adapter — pick one deliberately and document it.
• Retry budgets: cap retries as a fraction of total requests (say 10%), so retry traffic can’t dominate.
• Circuit breakers to stop retrying a dependency that’s clearly down. See 02_circuit_breakers_and_bulkheads.md.
• Propagate a deadline so a retry inside an already-expired budget doesn’t happen at all.

Where the policy lives

In the client/adapter layer, not scattered through business logic:

class PaymentClient:
    """One place that owns timeout, retry and breaker policy for this dependency."""
    def __init__(self, http: httpx.AsyncClient, breaker: CircuitBreaker): ...

Each dependency gets its own policy — a fast internal service and a slow third-party API should not share timeouts. Scattering @retry across service methods means you can’t reason about total latency or amplification.

Interview angle

• “How do you handle a flaky external API?” — timeouts first (connect, read, pool), then retry only transient classes with exponential backoff plus jitter and a capped attempt count, with idempotency keys for anything non-idempotent, all owned by the client layer rather than scattered.
• “Which errors do you retry?” — connection errors, timeouts, 502/503/504, and 429 honouring Retry-After. Not 4xx. Be careful with 500 — it’s often a deterministic bug, and retrying may duplicate a partial side effect.
• “Why jitter?” — without it, all clients that failed together retry together, producing synchronised waves that re-break the service on each recovery attempt. Jitter spreads them out; full jitter spreads hardest.
• “You added retries and the outage got worse. Why?” — retry amplification. Three retries at each of three layers is nine requests per user action, and that load is what pushes a slow service into a dead one. Retry at one layer, add a retry budget, and use a breaker.
• “A POST timed out. Do you retry?” — only with an idempotency key stable across attempts. A timeout means you don’t know whether it applied, so a blind retry can double-charge. Without a key, surface the uncertainty rather than guessing.
• “How do you pick timeout values?” — from the dependency’s p99, then budget downward from your own SLA. If your endpoint must answer in 3s and calls three services, they can’t each have a 3s timeout. Propagate the remaining deadline.