Circuit breakers and bulkheads

Two patterns that stop one failing dependency from consuming the resources the rest of your service needs. Retries protect against a blip; these protect against a sustained failure.

Circuit breaker

Stop calling a dependency that is clearly down, so you fail fast instead of paying the timeout on every request.

CLOSED ──failures exceed threshold──> OPEN
  ^                                    │
  │                              cooldown elapses
  │                                    v
  └───probe succeeds──────────── HALF-OPEN ───probe fails───> OPEN

class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown=30.0, half_open_max=1):
        self.state = "closed"
        self.failures = 0
        self.opened_at = 0.0

    async def call(self, fn, *args):
        if self.state == "open":
            if time.monotonic() - self.opened_at < self.cooldown:
                raise CircuitOpen                       # fail fast, no network call
            self.state = "half_open"                    # let one probe through

        try:
            result = await fn(*args)
        except TransientError:
            self._record_failure()
            raise
        else:
            self._record_success()
            return result

Why it matters

The value isn’t the failed calls you skip — it’s the resources you don’t tie up. With a 10-second timeout and no breaker, every request to a dead dependency holds a worker for 10 seconds. At modest traffic that exhausts your pool, and now requests that don’t touch the dead dependency start failing too. The breaker converts a 10-second hang into an instant rejection.

It also stops you hammering a service that’s trying to recover.

Getting it right

• Count rate, not just absolute failures. “5 failures” fires spuriously on a low-traffic endpoint. “50% of at least 20 requests in 10 seconds” is meaningful.
• Only transient failures should trip it. A 404 or a validation error is not the dependency being unhealthy — counting them opens the breaker for a bug in your request.
• One breaker per dependency, not per service. A shared breaker means a failing payment provider blocks calls to your search service.
• Half-open must be limited. Letting the full load through on the first probe re-kills a recovering service. One or two probes.
• Emit the state change as a metric. A breaker opening is an incident signal; if it opens silently you learn about it from users.

Libraries: pybreaker, or built into service meshes and API gateways (Envoy, Istio). Increasingly this belongs in the mesh rather than in every application.

Bulkheads

Named after ship compartments: isolate resources so flooding one section doesn’t sink the vessel.

The failure being prevented: one slow dependency consuming the shared pool. If all outbound calls share one connection pool or one thread pool, a dependency that goes slow occupies every slot, and unrelated requests starve.

# Per-dependency concurrency limits - one slow service can't consume everything
LIMITS = {
    "payments": asyncio.Semaphore(20),
    "search":   asyncio.Semaphore(50),
    "reports":  asyncio.Semaphore(5),      # slow and non-critical: strictly capped
}

async def call_payments(...):
    async with LIMITS["payments"]:
        return await payment_client.charge(...)

Applied at several levels:

The report endpoint capped at 5 concurrent is the pattern worth naming: a non-critical slow path gets a deliberately small allocation so it can never dominate.

Load shedding

The third member of this family. When you’re over capacity, reject some requests immediately rather than degrading for everyone.

if queue_depth > SHED_THRESHOLD:
    raise HTTPException(503, headers={"Retry-After": "5"})

Counterintuitive but correct: serving 70% of requests well beats serving 100% of them past the timeout, where nobody gets a usable response and you’ve spent the capacity anyway. Shed the lowest-priority traffic first — background jobs and analytics before user-facing requests.

How they compose

request
  -> load shedding      (are we over capacity at all?)
  -> bulkhead           (is this dependency's allocation free?)
  -> circuit breaker    (is this dependency known-bad?)
  -> timeout + retry    (attempt, bounded)
  -> fallback           (what do we return if it failed?)

Each layer handles a different failure duration: retries for a blip, breakers for a sustained outage, bulkheads for slowness, shedding for overload. Naming that progression is a strong answer. Fallbacks are in 03_fallbacks_and_degradation.md.

Interview angle

• “What does a circuit breaker actually buy you?” — resource protection. Without it, every request to a dead dependency holds a worker for the full timeout, exhausting the pool so unrelated requests fail too. The breaker turns a 10-second hang into an instant rejection and stops you hammering a recovering service.
• “How do you tune the threshold?” — on failure rate over a minimum request count, not absolute failures, or a low-traffic endpoint trips spuriously. Only count transient failures; a 404 isn’t the dependency being unhealthy.
• “What’s the risk with half-open?” — letting full traffic through on recovery immediately re-kills the service. Allow one or two probes, and only close the circuit after they succeed.
• “What is a bulkhead?” — resource isolation per dependency: separate connection pools or concurrency limits, so one slow dependency can’t consume the shared capacity that unrelated requests need.
• “You’re over capacity. Degrade everything or reject some?” — reject some. Serving 70% of requests well beats serving 100% past the timeout, where you spend the capacity and nobody gets a usable answer. Shed low-priority traffic first.
• “How do these fit together?” — different failure durations: retries for a transient blip, circuit breakers for a sustained outage, bulkheads for a slow dependency, load shedding for overload, fallbacks for what you return when all of it fails.