Cancellation, timeouts and error handling in asyncio

The area that separates people who have used asyncio from people who have shipped it. Everything here is about one question: when something goes wrong or takes too long, what actually stops, and what silently keeps running?

Cancellation is an exception, not a kill

task.cancel() does not stop a task. It schedules a CancelledError to be raised inside the coroutine at its next await point. A coroutine that never awaits again can never be cancelled.

task = asyncio.create_task(work())
task.cancel()
try:
    await task
except asyncio.CancelledError:
    ...  # the task finished cancelling

Two consequences people get wrong:

• A CPU-bound coroutine is uncancellable. No await, no delivery point. This is the same reason it blocks the loop.
• CancelledError inherits from BaseException, not Exception — since 3.8. A bare except Exception: will not swallow it, which is deliberate. A bare except: will, and that’s a bug.

Catch it only to clean up, then re-raise:

async def worker():
    try:
        await do_work()
    except asyncio.CancelledError:
        await flush_partial_state()   # cleanup
        raise                          # never swallow

Swallowing CancelledError breaks TaskGroup, asyncio.timeout() and graceful shutdown, because they all rely on cancellation actually propagating.

Timeouts: asyncio.timeout() over wait_for()

# Modern (3.11+) - a context manager, composes with anything
async with asyncio.timeout(5):
    data = await fetch()

# Deadline instead of duration
async with asyncio.timeout_at(loop.time() + 5):
    ...

On expiry the block’s task is cancelled and a TimeoutError is raised at the boundary. Since 3.11, asyncio.TimeoutError is an alias of the builtin TimeoutError — catch either.

asyncio.wait_for(aw, timeout) still works and is fine for wrapping a single awaitable, but asyncio.timeout() wraps a block of arbitrary code, which is what you usually want.

The mechanism worth being able to explain: the timeout cancels the inner task, then uncancels it (Task.uncancel(), 3.11+) so the cancellation stops at that block instead of leaking outward and killing the caller. That is why nested timeouts and TaskGroups compose correctly — each structured block owns its own cancellation.

shield — and why it’s usually the wrong tool

asyncio.shield(aw) stops cancellation from the caller reaching the inner awaitable. If the caller is cancelled, the caller still gets CancelledError, but the shielded work keeps running.

# The caller may be cancelled; the write must still land.
await asyncio.shield(persist_result(row))

Sharp edges:

• Shield only blocks cancellation propagating inward from the caller. inner.cancel() directly, or the loop shutting down, still cancels it.
• The shielded task is now unowned. If the caller goes away, nothing awaits the result and nothing observes its exception. You’ve created a background task with no supervisor.
• If the process exits before it finishes, it dies anyway — shield is not durability. Work that must happen belongs in a queue, an outbox row, or a workflow engine, not in a shielded coroutine. See ../13_architecture_design/16_transactional_outbox.md.

Reach for shield rarely, for bounded critical cleanup, and always await it somewhere in your shutdown path.

ExceptionGroup and except*

When concurrent siblings fail, more than one can fail at once, so 3.11 introduced a container for that.

try:
    async with asyncio.TaskGroup() as tg:
        tg.create_task(a())
        tg.create_task(b())
except* ValueError as eg:          # note the star
    for err in eg.exceptions:
        log.warning("bad value: %s", err)
except* ConnectionError as eg:
    ...

except* selects the matching leaves out of the group and can run more than one handler for a single raise. Plain except ExceptionGroup: also works if you want the whole thing.

TaskGroup always raises an ExceptionGroup (or BaseExceptionGroup), even for a single failure — that’s the API contract, and it surprises people migrating from gather.

Failure semantics, side by side

gather’s behaviour is the classic production bug: the first exception surfaces, your handler moves on, and the other five requests are still in flight against a service you just decided was broken.

Default to TaskGroup. Use gather(return_exceptions=True) when partial failure is genuinely acceptable and you want to inspect every outcome — a health-check fan-out, for instance. Detail in 36_taskgroup_structured_concurrency.md.

Fire-and-forget loses your exceptions

asyncio.create_task(background_work())   # bug on two counts

1. The loop keeps only a weak reference to a task. With no strong reference, it can be garbage-collected mid-flight — the “Task was destroyed but it is pending!” warning.
2. If it raises and nobody awaits it, the exception surfaces only when the task is GC’d, as a “Task exception was never retrieved” message. Not in your error tracker, and not at the time it happened.

Minimum viable fix:

_background: set[asyncio.Task] = set()

def spawn(coro) -> asyncio.Task:
    task = asyncio.create_task(coro)
    _background.add(task)                       # strong ref
    task.add_done_callback(_background.discard)
    task.add_done_callback(_log_if_failed)
    return task

def _log_if_failed(task: asyncio.Task) -> None:
    if not task.cancelled() and task.exception():
        log.exception("background task failed", exc_info=task.exception())

Better: don’t fire-and-forget. Own the task in a TaskGroup scoped to the request, or push the work to a real queue.

Graceful shutdown

async def main():
    async with asyncio.TaskGroup() as tg:      # owns the workers
        tg.create_task(consume())
        tg.create_task(serve())
    # exiting the block waits for both, or cancels both on failure

When you can’t wrap everything in a group — an existing service, say — the shutdown path is: stop accepting new work, cancel outstanding tasks, then await them with a bounded timeout so a stuck task can’t hang the process.

for task in tasks:
    task.cancel()
results = await asyncio.gather(*tasks, return_exceptions=True)

return_exceptions=True matters here: without it, the first CancelledError aborts the gather and you stop waiting for the rest to finish cleaning up.

Interview angle

• “What does task.cancel() actually do?” — schedules CancelledError into the coroutine at its next await. It’s a request, not a kill. A coroutine with no further await cannot be cancelled, and one that catches and swallows CancelledError breaks every structured-concurrency construct above it.
• “Why does CancelledError derive from BaseException?” — so routine except Exception: handlers don’t accidentally swallow a cancellation and keep a task alive that the runtime is trying to tear down.
• “wait_for vs asyncio.timeout?” — wait_for wraps one awaitable; asyncio.timeout() (3.11+) wraps a block, composes with nesting, and uses uncancel() so the cancellation is scoped to that block rather than leaking to the caller. Prefer the context manager.
• “One of five gather tasks raises — what happens to the other four?” — they keep running, unsupervised. gather surfaces the first exception and does not cancel siblings. TaskGroup cancels them and gives you an ExceptionGroup. This is the single best reason to default to TaskGroup.
• “When would you use shield?” — bounded critical work that must survive caller cancellation, and only when you still await it during shutdown. It doesn’t survive process exit, so it is not a durability mechanism — that’s an outbox or a workflow engine.
• “What’s wrong with asyncio.create_task(f()) on its own?” — no strong reference, so it can be GC’d mid-flight, and its exception is never retrieved, so failures vanish. Keep a reference set plus a done-callback, or own it in a TaskGroup.
• “How do you catch one error type out of a concurrent fan-out?” — except* ValueError as eg: and iterate eg.exceptions. Multiple except* clauses can fire for a single raise, because more than one sibling can fail.