Async I/O and background work

Two decisions that get conflated: how to run several I/O calls concurrently inside a request, and when work should leave the request entirely.

Inside the request: concurrency

# Sequential - latency is the SUM
user = await users.get(uid)          # 120ms
prefs = await prefs.get(uid)         # 80ms
history = await history.get(uid)     # 200ms   -> 400ms total

# Concurrent - latency is the MAX
async with asyncio.TaskGroup() as tg:
    user_t = tg.create_task(users.get(uid))
    prefs_t = tg.create_task(prefs.get(uid))
    history_t = tg.create_task(history.get(uid))   # -> 200ms total

Independent calls should never run sequentially. It’s the most common and most easily fixed latency problem in an async codebase, and await in a loop is the usual shape of it.

TaskGroup (3.11+) over gather: on failure it cancels siblings and raises an ExceptionGroup, so you don’t leak tasks that keep running against a service you’ve already decided is broken. See ../../backend/04_async_concurrency/12_taskgroup_structured_concurrency.md.

Use gather(return_exceptions=True) deliberately when partial success is the goal — see ../02_resilience/03_fallbacks_and_degradation.md.

Bound the fan-out

sem = asyncio.Semaphore(10)

async def fetch_one(item):
    async with sem:                      # acquire INSIDE the task, not around creation
        return await client.get(item.url)

async with asyncio.TaskGroup() as tg:
    tasks = [tg.create_task(fetch_one(i)) for i in items]   # 10_000 items is fine

Creating 10,000 tasks at once is fine — they’re cheap. Making 10,000 simultaneous HTTP calls is not: you exhaust the connection pool, trip the provider’s rate limit, and starve everything else. The semaphore must be acquired inside the coroutine; wrapping task creation instead serialises the whole thing.

Never block the event loop

One synchronous call stalls every coroutine in the process.

# Wrong - blocks the loop for the entire duration
result = requests.get(url)               # sync HTTP
data = pd.read_csv(huge_file)            # sync I/O + CPU
hashed = bcrypt.hashpw(pw, salt)         # CPU, ~100ms

# Right
result = await client.get(url)                        # async client
data = await asyncio.to_thread(pd.read_csv, path)     # blocking I/O -> thread
hashed = await loop.run_in_executor(process_pool, bcrypt.hashpw, pw, salt)   # CPU -> process

The symptom of a blocked loop is distinctive: p99 latency rises across every endpoint at once, including ones that do nothing. Unrelated endpoints degrading together is the fingerprint. See ../../backend/04_async_concurrency/14_run_in_executor_to_thread.md.

Leaving the request: background work

Different question. Move work out of the request when:

• The user doesn’t need the result to proceed (email, webhook, indexing).
• It’s slow enough to blow the latency budget.
• It must survive a failure and be retried.
• It’s scheduled rather than triggered.

The options, and what each guarantees

The line that matters: the first two are in-process and die with it. Using BackgroundTasks to send a payment confirmation means a deploy mid-request silently loses it. If losing the work is unacceptable, it needs a broker.

# Acceptable: metrics, best-effort cache warm
background_tasks.add_task(track_event, user_id, "viewed")

# NOT acceptable: anything the business needs to have happened
await queue.enqueue("send_invoice", order_id=order.id)

The enqueue-after-commit trap

# Wrong - the worker may start before the transaction commits, or the row may never exist
await session.commit()
send_email.delay(order.id)         # if this fails, no email, no record of the failure

Two failure modes: enqueue before commit and the worker can read a row that doesn’t exist yet or gets rolled back; enqueue after commit and the enqueue itself can fail, leaving committed state with no job.

The correct fix is the transactional outbox: write the job to an outbox table in the same transaction, and let a relay publish it. Atomic by construction. See ../../backend/13_architecture_design/16_transactional_outbox.md.

Workers need the same discipline

• Idempotent — at-least-once delivery means every job may run twice.
• Bounded — job timeouts, or one stuck job holds a worker forever.
• Dead-letter queue after N failures, so a poison message doesn’t block the queue.
• Monitored — queue depth and oldest-message age are the two alerts that matter.

Queue depth rising is the earliest signal that consumers can’t keep up, and it precedes user-visible failure by a comfortable margin.

Choosing

Does the caller need the result now?
  yes -> in-request, concurrent, bounded, with timeouts
  no  -> Is losing it acceptable?
           yes -> fire-and-forget task
           no  -> Is it multi-step with compensation?
                    yes -> workflow engine (Temporal)
                    no  -> queue (Celery / Kafka), enqueued via outbox

Interview angle

• “Three independent API calls in one endpoint. How do you run them?” — concurrently with TaskGroup, so latency is the slowest rather than the sum, each with its own timeout under the overall budget. TaskGroup over gather because it cancels siblings on failure instead of leaking tasks.
• “You need to call an API for 10,000 items. Approach?” — create the tasks freely but bound concurrency with a semaphore acquired inside each coroutine. Unbounded fan-out exhausts the connection pool and trips rate limits; wrapping task creation in the semaphore accidentally serialises everything.
• “Every endpoint’s p99 went up, including ones doing nothing. Why?” — something is blocking the event loop. Find the sync call — a sync HTTP client, file I/O, or CPU work like password hashing — and move it to a thread or process.
• “When do you use FastAPI BackgroundTasks versus Celery?” — BackgroundTasks runs in-process and dies with it, so only for work you can afford to lose. Anything the business needs to have happened requires a broker with durability and retries.
• “You commit, then enqueue a job, and the enqueue fails. Now what?” — that’s the dual-write problem. Use a transactional outbox: write the job in the same transaction as the state change and let a relay publish it, so the two can’t diverge.
• “What do you monitor on a queue?” — depth and oldest-message age. Both rise before users notice, which makes them the earliest actionable signal that consumers can’t keep up.