Resilient orchestration across services

Coordinating several dependencies where any of them can fail. This is the MINT “how do you handle retries if a microservice dies” question, and the honest answer has three layers: bound the call, order the work so failure is cheap, and compensate when it isn’t.

Dependent vs independent calls

The first question is whether calls can run concurrently.

# Independent - fan out, bounded, partial failure tolerated
async with asyncio.TaskGroup() as tg:
    user_t = tg.create_task(users.get(uid))
    prefs_t = tg.create_task(prefs.get(uid))

# Dependent - sequential, because B needs A's output
user = await users.get(uid)
account = await accounts.get(user.account_id)

Sequential chains multiply latency and failure probability. Three calls at 99.9% each gives 99.7% end to end; ten gives 99%. That arithmetic is worth saying out loud — it’s the argument for reducing chain depth rather than making each link more reliable.

If calls are logically dependent but you control both services, consider whether one API could return both — the most effective resilience fix is often removing a hop.

Order operations so failure is cheap

Free advice that avoids most compensation work: do the reversible, cheap and likely-to-fail things first; do the irreversible thing last.

# Wrong: charge, then discover the item is gone
await payments.charge(...)
await inventory.reserve(...)      # fails -> now you must refund

# Right: validate and reserve, then charge
reservation = await inventory.reserve(...)    # reversible
await payments.charge(...)                    # irreversible, last
await inventory.confirm(reservation)

Validation before side effects, reservations before charges, external calls before local commits where possible. You can’t always achieve it, but each step you reorder is a compensation you don’t have to write.

Sagas — when you can’t avoid distributed state

There’s no distributed transaction across microservices, so a multi-step process that fails partway needs compensating actions that semantically undo completed steps.

async def book_campaign(order: Order) -> Result:
    completed: list[Compensation] = []
    try:
        res = await budget.reserve(order.amount, key=order.id)
        completed.append(lambda: budget.release(res.id, key=order.id))

        charge = await payments.charge(order, key=order.id)
        completed.append(lambda: payments.refund(charge.id, key=order.id))

        await campaigns.activate(order, key=order.id)
        return Result.ok()

    except Exception:
        for compensate in reversed(completed):        # unwind in reverse
            await run_with_retry(compensate)          # compensations must also retry
        raise

Four things that make this real rather than a diagram:

• Compensations run in reverse order.
• Compensations can fail too, so they need their own retries and a dead-letter path. A compensation that fails silently leaves permanently inconsistent state — this is the worst failure mode in the whole pattern.
• Everything is keyed by order.id so retries at any level are idempotent.
• Compensation is semantic, not a rollback. You can’t un-send an email; you send a correction. You can’t un-charge; you refund, and the customer sees both lines.

Orchestration vs choreography: a central orchestrator holding the state machine is easier to debug and to reason about; event-driven choreography couples less but makes “where is this order” genuinely hard to answer. For anything with money in it, prefer orchestration. See ../../backend/13_architecture_design/15_event_driven_saga.md.

The state has to be durable

An in-process saga dies with the process, leaving reserved budget and a charge with nothing to complete or unwind them.

Temporal is the strongest answer when the process spans minutes or days: workflow state is durable, retries and timeouts are declarative, and compensation is a normal code path rather than a hand-rolled unwind stack. See ../../backend/10_message_queues/temporal/03_temporal_vs_celery.md.

The transactional outbox solves the narrower and very common problem of “commit to my database and publish an event, atomically”. Write the event to an outbox table in the same transaction as the state change; a relay publishes it afterwards. Without it you get the dual-write problem: the commit succeeds and the publish fails, or vice versa. See ../../backend/13_architecture_design/16_transactional_outbox.md.

Idempotency everywhere

At-least-once delivery is the norm, so every step must tolerate being run twice. This is the precondition that makes retries and compensation safe, and it’s the single most important property in the whole design.

async def reserve(order_id: UUID, amount: Decimal) -> Reservation:
    existing = await repo.find_by_key(order_id)
    if existing:
        return existing                   # replay returns the same result
    ...

Natural keys where possible, explicit idempotency keys otherwise, with the dedupe record written in the same transaction as the effect.

Observability

A distributed process that fails needs to be answerable: which step, on which attempt, with what input?

• Propagate a correlation ID through every service and every retry.
• Log each step transition with the saga/workflow ID.
• Distributed tracing so the chain is one trace rather than seven disconnected logs.
• Alert on stuck instances — a saga that started and neither completed nor compensated is invisible unless you look for it.

That last one is the gap in most implementations. Failures that loop are noisy; sagas that simply stopped are silent.

Interview angle

• “How do you handle retries if a microservice dies mid-process?” — bound each call with timeouts and idempotent retries, keep the process state durable so it survives a restart, and compensate completed steps if it can’t proceed. Name the durability mechanism: a workflow engine, or a saga state table plus outbox.
• “What is a saga?” — a sequence of local transactions where each step has a compensating action, unwound in reverse on failure. Compensation is semantic, not a rollback: you refund rather than un-charge, and the customer sees both entries.
• “What goes wrong with sagas in practice?” — compensations failing silently, which leaves permanently inconsistent state. They need their own retries and dead-letter path. Second: in-memory saga state that dies with the process.
• “Orchestration or choreography?” — orchestration for anything financial or auditable, because the state machine is in one place and you can answer “where is this order”. Choreography couples services less but makes debugging genuinely hard.
• “Why is idempotency the precondition?” — at-least-once delivery means every step may run twice, so retries and compensations are only safe if repeating them is harmless. Key by the business operation and write the dedupe record in the same transaction as the effect.
• “How would you reduce failure probability in a chain of calls?” — shorten the chain. Three 99.9% calls give 99.7%, ten give 99%. Merging two hops into one API does more than tuning either link, and parallelising independent calls cuts latency without changing the arithmetic.