Durable execution and human-in-the-loop

Where AI agents meet ordinary distributed-systems engineering. This is the strongest ground for a backend engineer in an AI interview — the problems are ones you already know how to solve.

The problem

A naive agent is a function call. It holds state in local variables and dies with its process.

That breaks the moment you need anything real:

• A deploy mid-run loses the conversation.
• A worker crash loses an hour of work.
• An approval step means waiting hours or days — you can’t hold a process open.
• A tool times out and there’s no way to resume from just after it.
• A user asks what the agent did last Tuesday and there’s no record.

An agent needs to be a durable process, not a function call.

Checkpointing

Persist state after each step so execution can resume from the last good point.

from langgraph.checkpoint.postgres import PostgresSaver

graph = builder.compile(checkpointer=PostgresSaver.from_conn_string(DSN))
config = {"configurable": {"thread_id": "run-42"}}

graph.invoke({"messages": [msg]}, config)   # starts, or resumes mid-flight

What a checkpoint must contain: the message history, any scratch state, which node is next, and enough metadata to reconstruct the run. What it must not contain: open connections, file handles, or anything unserialisable — the usual constraint on resumable work.

Beyond crash recovery, checkpointing gives you time-travel debugging: rewind to any step, change state, re-run from there. For a non-deterministic system that’s a materially better debugging story than reading logs.

Human-in-the-loop

The pattern that requires durability. You cannot hold a process open while someone approves a refund tomorrow morning.

from langgraph.types import interrupt, Command

def approve(state):
    decision = interrupt({
        "action": "refund",
        "order_id": state["order_id"],
        "amount": state["amount"],
        "reason": state["reason"],
    })
    if decision["approved"]:
        return {"result": execute_refund(state)}
    return {"result": f"Denied: {decision.get('note', 'no reason given')}"}

# Hours later, different process, from a web handler:
graph.invoke(Command(resume={"approved": True}), config)

interrupt persists state and returns control. The run resumes from anywhere holding the thread_id.

Where to put the gate

Approve-before-acting is the one to name by default. Reviewing the plan before execution is the cheapest intervention — catching a wrong approach before twenty tool calls beats correcting the output.

Designing the approval payload

Give the reviewer enough to decide without re-deriving the agent’s reasoning: the action and its arguments, why the agent chose it, what it will affect, and what happens if denied. An approval UI showing {"tool": "refund_order", "args": {...}} and nothing else forces the reviewer to guess.

The distributed-systems layer

Agents hit every classic problem, and the classic answers apply.

Idempotency. Agents retry. A retried send_payment must not charge twice. Client-supplied idempotency keys, deduped server-side. See ../../backend/13_architecture_design/18_idempotency_keys.md.

Timeouts and retries. Per tool call, with exponential backoff and jitter; distinguish 429 from 5xx from 4xx. Cap total wall-clock for the run, not just per call. See ../../system_design/02_resilience/.

Compensation. A multi-step agent that fails at step 7 may need to undo steps 1-6. That’s a saga, and it’s the right vocabulary to use. See ../../backend/13_architecture_design/15_event_driven_saga.md.

Exactly-once effects. You can’t get exactly-once delivery, so make effects idempotent and reconcile. Same answer as any message-driven system.

LangGraph or a workflow engine

A real question once agents run long and have side effects.

A defensible position: use LangGraph for the agent loop, and a workflow engine for the long-running business process that contains it. The agent decides what to do; Temporal guarantees the multi-step process completes, retries and compensates.

For an agent orchestrating a refund across payment, inventory and notification services, that separation is exactly right — and saying so demonstrates you see the agent as one component in a system rather than the system itself. See ../../backend/10_message_queues/temporal/03_temporal_vs_celery.md.

Operating it

• Thread ID scheme — how do you find a run later? Usually a conversation or case ID.
• Checkpoint retention — they accumulate. Set a TTL; keep an audit summary longer than the full state.
• Sensitive data in checkpoints. They persist message history, which may contain PII. Encrypt at rest, honour deletion requests, and think about it before an auditor does.
• Version skew. A run checkpointed under graph v1 resuming under v2 may hit nodes that no longer exist. Version your graphs and decide the policy: drain, or migrate.
• Stuck runs. Something interrupted for approval and never resumed needs a timeout and an escalation path.

Version skew and orphaned interrupts are the two that bite in production and rarely come up in tutorials.

Interview angle

• “Why does an agent need durable execution?” — approvals take hours, deploys happen mid-run, workers crash, and audits ask what happened. A function call dies with its process; a checkpointed graph resumes from the last good step.
• “How do you implement an approval step?” — interrupt and persist state, surface the pending action to a human with enough context to decide, and resume by thread ID when they respond. Without durable state you’d be building a state machine and a job queue yourself.
• “An agent retried and charged the customer twice. Fix?” — idempotency keys on any tool with side effects, deduped server-side. Agents retry by design; this is an ordinary distributed-systems problem.
• “An agent fails at step 7 of 10 with side effects already applied. What now?” — compensating transactions, i.e. a saga. Either design each step to be undoable, or structure the process so effects commit only at the end.
• “LangGraph or Temporal?” — both, at different levels. LangGraph for the agent loop where LLM ergonomics matter; a workflow engine for the long-running business process containing it, where retries, compensation and multi-week durability are first-class.
• “What goes wrong operationally with checkpointing?” — unbounded checkpoint growth, PII persisted in message history, graph version skew when a run resumes under new code, and interrupted runs that nobody ever resumes. All need explicit policies.