Resilience patterns for service-to-service calls

In a microservice system every request fans out into N network calls, and the failure mode that kills systems is rarely “dependency down” — it’s “dependency slow.” Slow calls hold connections, fill thread pools and event loops, and back pressure up the call graph until healthy services fall over too: a cascading failure. These patterns exist to convert unbounded slowness into fast, bounded failure.

Basics of timeouts/retries/backoff and the breaker state machine are in ../../system_design/02_resilience/01_timeouts_retries_backoff.md and ../../system_design/02_resilience/02_circuit_breakers_and_bulkheads.md; this note is how they compose in a service mesh of your own services.

Timeouts — with a budget, not per hop

Every call gets a timeout; no exceptions. The senior detail is the deadline budget: if the edge promises 2 s, inner hops must divide what remains, or an inner retry can happily succeed after the caller has already given up.

edge (2000ms) → checkout (1500ms) → pricing (400ms) → fx-rates (150ms)

Propagate the remaining budget (X-Deadline header / gRPC deadlines do this natively) and refuse work when it’s already exhausted. Connect timeout short (~1 s, it’s LAN), read timeout sized to the dependency’s p99 — not its average.

Retries — the pattern that causes outages

Retries repair transient failures (dropped connection, one bad pod) and amplify persistent ones: a dependency at 100% CPU receiving 3× traffic from everyone’s retries stays down — a retry storm.

Rules that survive production:

• Retry only idempotent operations (../13_architecture_design/18_idempotency_keys.md) and only retryable errors (connect errors, 502/503/504, timeouts maybe; never 4xx).
• Exponential backoff with jitter — synchronized retries are a self-inflicted thundering herd.
• Cap attempts (2–3) and use a retry budget: if >10% of recent requests are retries, stop retrying — degrade instead.
• One layer retries. Client library, mesh sidecar (05_service_mesh.md), and gateway each retrying 3× = 27 attempts per user request. Decide where retries live and disable the rest.

Circuit breaker — stop calling the dead

After enough consecutive failures, fail immediately without dialing the dependency; probe occasionally; close when it recovers. Protects your resources (no connections parked on a corpse) and their recovery (no hammering). One breaker per dependency, shared by all callers in the process.

Bulkheads — one failing dependency must not sink the ship

Named after hull compartments: partition resources per dependency so exhaustion is contained.

import asyncio, httpx

# separate pools per dependency — slow "recs" can't starve "payments"
payments = httpx.AsyncClient(base_url="http://payments",
                             limits=httpx.Limits(max_connections=50), timeout=2.0)
recs = httpx.AsyncClient(base_url="http://recs",
                         limits=httpx.Limits(max_connections=10), timeout=0.8)

_recs_gate = asyncio.Semaphore(20)   # cap in-flight recs work explicitly

async def get_recommendations(user_id: int) -> list[dict]:
    async with _recs_gate:
        r = await recs.get(f"/users/{user_id}")
        r.raise_for_status()
        return r.json()

The same idea at other layers: separate worker pools for critical vs best-effort jobs, separate DB connection pools per workload, k8s resource limits per deployment. The anti-pattern is one shared pool where the slowest dependency eventually owns every slot.

Load shedding and backpressure — protect yourself from callers

The inbound mirror of the outbound patterns: past a concurrency/queue limit, reject early with 429/503 + Retry-After instead of queueing into latency collapse. A request served in 8 s is often worth less than a fast, honest failure — and the retry-after lets well-behaved clients back off. Rate limiting algorithms: ../13_architecture_design/17_rate_limiting_algorithms.md; queue-depth thinking: bounded queues everywhere, because unbounded queues turn overload into memory exhaustion plus infinite latency.

Fallbacks — degrade on purpose

When the call fails fast, do something better than a 500 — in order of preference: cached last-known-good, static default (empty recommendations, generic shipping estimate), reduced feature (“reviews unavailable”), queued-for-later (accept the write, process async). Fallbacks are product decisions: falling back silently on a payments authorization is a bug, not resilience. Decide per endpoint what honest degradation looks like (../../system_design/02_resilience/03_fallbacks_and_degradation.md).

Composing them — order matters

Per attempt, outside-in: breaker → retry loop → per-attempt timeout, inside a bulkhead, with a fallback wrapping the whole thing.

import pybreaker
from tenacity import retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type

pricing_breaker = pybreaker.CircuitBreaker(fail_max=5, reset_timeout=30)

@retry(stop=stop_after_attempt(3),
       wait=wait_exponential_jitter(initial=0.1, max=2.0),
       retry=retry_if_exception_type((httpx.ConnectError, httpx.ReadTimeout)),
       reraise=True)
async def _call_pricing(order_id: int) -> dict:
    r = await pricing_client.get(f"/quote/{order_id}")   # client owns the timeout
    r.raise_for_status()
    return r.json()

async def get_quote(order_id: int) -> dict:
    try:
        return await pricing_breaker.call_async(_call_pricing, order_id)
    except (pybreaker.CircuitBreakerError, httpx.HTTPError):
        return cached_quote(order_id)                     # explicit, logged fallback

Breaker outside retry: three failed attempts count once toward opening, and an open circuit skips the retry loop entirely.

Code vs sidecar — where should these live?

Mesh handles the mechanical layer uniformly; fallbacks and idempotency are always yours. If a mesh retries, your code shouldn’t (05_service_mesh.md).

Health checking and pulling bad instances out of rotation belong to the platform: LB outlier ejection, k8s readiness probes (../17_kubernetes/04_probes_and_hpa.md), discovery-level health (01_service_discovery.md).

Common pitfalls

• Timeouts without a deadline budget — inner work completes for callers that already left.
• Retrying non-idempotent POSTs — duplicate charges; fix with idempotency keys first, retries second.
• Retry stacking across client + mesh + gateway.
• Breaker per call site instead of per dependency — 40 half-blind breakers none of which ever opens.
• Fallback that hides a broken money path — degrade loudly (log + metric + alert) or not at all.

Interview angle

• “A downstream service got slow — walk me through what happens to yours.” — connection/loop saturation → cascading failure; then the fix stack: timeout budget, breaker, bulkhead, shed load.
• “How do retries make outages worse, and what prevents it?” — retry storms; jitter, caps, retry budgets, idempotency, single-layer ownership.
• “Circuit breaker vs retry vs bulkhead — one sentence each?” — retry = survive blips; breaker = stop calling the dead; bulkhead = contain the blast radius.
• “Service mesh gives you retries and timeouts — why still write resilience code?” — fallbacks and idempotency are business logic; and someone must ensure the layers don’t stack.