Fallbacks, partial failure and graceful degradation

What you return when a dependency has failed and you’ve exhausted retries. The engineering is easy; the hard part is deciding, in advance, what “degraded but useful” means for each feature.

The fallback ladder

Ordered by how much value the user still gets:

Serving stale data is usually better than serving nothing — but it must be a decision, not an accident. A cached exchange rate from five minutes ago is fine; a cached account balance from five minutes ago may be a compliance problem.

That distinction is the interview point: the right fallback is a business question, and different per feature. Ask which, rather than proposing one universally.

Stale-while-revalidate

The most useful caching shape for resilience:

async def get_price(sku: str) -> Price:
    cached = await cache.get(sku)
    if cached and not cached.is_stale:
        return cached.value

    try:
        fresh = await pricing_service.get(sku, timeout=1.0)
        await cache.set(sku, fresh, ttl=300, stale_ttl=3600)
        return fresh
    except (TimeoutError, ServiceUnavailable):
        if cached:                                   # stale beats nothing
            metrics.increment("price.served_stale")
            return cached.value
        raise

Two TTLs: a fresh window where you serve without asking, and a longer stale window where you’d prefer fresh but will serve old data if the source is down. The dependency being down degrades freshness rather than availability.

Emit a metric when you serve stale. Otherwise a dependency can be down for hours while dashboards look healthy — the failure is invisible precisely because the fallback worked.

Partial failure

When you fan out to several sources, all-or-nothing is usually the wrong contract.

async def search_all(query: str) -> SearchResults:
    async with asyncio.TaskGroup() as tg:
        tasks = {
            name: tg.create_task(with_timeout(provider.search(query), 0.8))
            for name, provider in PROVIDERS.items()
        }
    # gather results, recording which sources failed
    results, failed = [], []
    for name, task in tasks.items():
        try:
            results.extend(task.result())
        except (TimeoutError, ProviderError):
            failed.append(name)

    return SearchResults(items=results, degraded_sources=failed)

Note TaskGroup cancels siblings on an unhandled exception, so each call needs its own error handling if partial success is the goal — otherwise one failure takes the whole fan-out down. See ../../backend/04_async_concurrency/12_taskgroup_structured_concurrency.md.

Make degradation visible in the response. degraded_sources lets the UI say “results may be incomplete” instead of silently showing less. Users tolerate a stated limitation far better than a silent one.

Deciding what’s acceptable

Classify each dependency before the incident, not during it:

An optional dependency must never fail the request. A common real bug: an analytics call inside the request path without a timeout, so an analytics outage takes down checkout. Fire it async, cap it hard, and swallow its errors.

asyncio.create_task(track_event(...))     # never awaited on the request path

Write this classification down. During an incident is the wrong time to discover that nobody agreed whether recommendations are critical.

Error budgets

The framing that makes “acceptable failure” concrete. A 99.9% availability target permits roughly 43 minutes of downtime per month — that’s the budget.

It converts reliability from a vague aspiration into a number you spend: budget remaining means you can ship risky changes, budget exhausted means you stop feature work and fix reliability. See ../../backend/15_observability/12_slo_sli_sla.md.

The senior version of “how reliable should this be”: not 100%. Chasing nines past what users notice costs more than it returns, and the budget makes that trade explicit.

Testing it

Fallback paths are the least-tested code in most systems, and they run during your worst incidents.

• Unit-test the fallback branch directly — inject a failing client and assert the degraded response.
• Fault injection: a test proxy (toxiproxy) or a mesh fault rule that adds latency and errors.
• Chaos testing in staging: kill a dependency and confirm the system degrades as designed rather than cascading.
• Game days: rehearse an outage with the team. Frequently reveals that the fallback was never wired up.

The recurring discovery: the fallback exists, and a config flag disables it, or it calls the same dead dependency.

Interview angle

• “A dependency is down and retries are exhausted. Now what?” — walk the ladder: cached value, partial result, default or heuristic, reduced feature, queue for later, honest error. Which one is correct is a business decision per feature — stale prices are fine, stale balances may not be.
• “How do you serve stale data safely?” — two TTLs, fresh and stale. Serve fresh inside the first window, attempt refresh after it, and fall back to stale if the source is down. Emit a metric when you serve stale, or the outage stays invisible while dashboards look fine.
• “Three providers, one is down. What do you return?” — partial results with the failed sources named in the response, so the UI can say results may be incomplete. Users accept a stated limitation; they don’t accept silently wrong output.
• “An analytics outage took down checkout. What went wrong?” — an optional dependency was on the critical path, awaited and without a hard timeout. Classify dependencies as critical, important or optional up front, and make optional calls fire-and-forget with errors swallowed.
• “How do you decide what reliability to target?” — an error budget from the SLO. 99.9% is about 43 minutes a month; while budget remains you ship, when it’s exhausted you stop and fix. It makes “not 100%” an explicit, defensible position.
• “How do you know the fallback works?” — test it directly with an injected failing client, then fault injection and chaos testing in staging. Fallback code is the least-exercised path and it runs during your worst incidents.