Celery Canvas: Chord, Group, Chain — and Their Pitfalls

Celery’s “Canvas” is the workflow API: chain, group, chord, chunks, map, starmap. They’re powerful and have surprising failure modes. Knowing the gotchas is a senior signal.

Building blocks

from celery import chain, group, chord, signature

# Chain: pipeline — output of one is input to next
chain(fetch.s(url), parse.s(), store.s())()

# Group: parallel fan-out
group(send_email.s(email) for email in emails)()

# Chord: group + callback. Callback runs when ALL group tasks complete.
chord(
    [process_chunk.s(c) for c in chunks],
    combine_results.s(),
)()

.s(args) creates a signature — a task call you can chain together but haven’t enqueued yet. signature(...) is the verbose form.

Chain pitfalls

chain(fetch.s(url), parse.s(), store.s())()

Looks like Unix pipeline. Watch out:

1. Failure stops the chain. If parse fails, store never runs. Sometimes that’s what you want; sometimes you wanted “best-effort, store partial results.”

2. Each step is a separate task in the broker. A chain of 5 steps = 5 enqueues, 5 dequeues, 5 result-backend writes. Adds latency. For tightly-coupled work, one task with internal logic is cheaper.

3. The implicit data flow can balloon. Each step’s output becomes the next step’s input — serialized through the broker. Don’t pass big blobs through a chain; pass references.

4. Chained signatures vs explicitly chained.

# These look similar; semantics differ
chain(a.s(), b.s(), c.s())()              # task a, then b(a's result), then c(b's result)
a.apply_async(link=b.s())                 # task a; on success, b enqueued (a's result discarded by b unless declared)

link= is the older syntax. Stick with chain().

Group pitfalls

group(send_email.s(e) for e in emails)()

1. Group must complete to get the result. result.get() blocks until all tasks finish. For 10k emails, that’s a long block. Use result.children to check status incrementally.

2. Memory in the result backend. Each task’s result is stored. 10k tasks × 1KB results = 10MB in Redis just for the group. Set ignore_result=True if you don’t need them:

group(send_email.s(e).set(ignore_result=True) for e in emails)()

3. Group inside a chain. Chains can contain groups; what you get is more complex than you’d think:

chain(start.s(), group(do_stuff.s(i) for i in range(10)), finish.s())()

The group becomes a chord automatically — finish runs after all 10 do_stuff complete. Read the result type carefully.

Chord pitfalls — the famous trap

chord([process_chunk.s(c) for c in chunks], combine_results.s())()

1. The callback only runs when all body tasks succeed. If ANY task in the group fails (after retries exhausted), the chord enters error state and combine_results may never run. Common bug: chord stuck for hours because one task is in a retry loop.

Mitigation: chord_unlock_max_retries, task_chord_error_callback, or design body tasks to never raise (catch and return error markers).

2. Chord uses the result backend extensively. It needs to know “have all body tasks finished?” — polled or counted in Redis. Heavy chord usage causes result-backend load spikes.

3. The chord_unlock task polls the body tasks’ state. On slow backends, this polling itself causes issues. On Redis, fine. On the deprecated AMQP result backend, unusable.

4. Race condition on chord completion. Historically, chord callbacks could fire too early under specific conditions (the unlock task evaluated state during a transient inconsistency). Fixed in modern Celery (4.x+), but be wary on older versions.

5. Chord doesn’t preserve order. The callback receives a list of results, but the order isn’t guaranteed across all backends — assume unordered. If order matters, embed an index in each result.

@app.task
def process_chunk(chunk, index):
    return {"index": index, "value": do_work(chunk)}

@app.task
def combine_results(results):
    sorted_results = sorted(results, key=lambda r: r["index"])
    return [r["value"] for r in sorted_results]

chord([process_chunk.s(c, i) for i, c in enumerate(chunks)], combine_results.s())()

Backend-specific behavior

For chord-heavy workflows, Redis backend is the default expectation.

Group + DLQ interaction

If one task in a group fails permanently and ends up in the DLQ, the rest of the group can complete fine — but the chord callback (if any) waits indefinitely. You need a “failure callback” that resolves the chord with partial results:

@app.task
def combine_results_or_partial(results):
    successful = [r for r in results if not isinstance(r, dict) or r.get("ok")]
    return summarize(successful)

chord(
    [process.s(item).set(retry_kwargs={"max_retries": 3}) for item in items],
    combine_results_or_partial.s(),
).apply_async(link_error=on_chord_failure.s())

When to skip Canvas entirely

For complex workflows, consider:

• Temporal — durable workflows, much cleaner branching, retries, compensations. See 10_message_queues/temporal/.
• AWS Step Functions — state machine in JSON; if you’re on AWS and the flow fits a state machine.
• Hand-rolled orchestration — a “coordinator task” that explicitly enqueues child tasks and tracks state in your DB.

Canvas works for simple fan-out (group) and pipelines (chain). For “step 1 succeeds, then run steps 2-5 in parallel, then step 6 if any of those produces X” — you’re better off with Temporal.

Common bugs

• Chord stuck because one body task is retrying forever. Cap retries; route exhausted tasks to DLQ so the chord can complete.
• Out-of-memory worker on big groups. Each task’s pickled result lives in the result backend; combine_results loads them all. Stream / paginate large fan-outs.
• Recursive groups inside chains. Hard to reason about; harder to debug.
• Mixing ignore_result=True with chord. Chord needs to know body tasks finished. Don’t ignore results inside a chord body.
• Chord callback fires on partial results. Old bug; ensure modern Celery; test with deliberately slow tasks.

Interview angle

• “What’s the difference between chain, group, and chord?” — chain: pipeline (a → b → c). group: parallel fan-out. chord: group + callback (callback runs after all body tasks succeed).
• “What goes wrong with chord if one task fails?” — chord enters error state; the callback never runs. Hangs until you intervene. Mitigation: cap retries on body tasks, route failures to DLQ so the chord can finalize, or handle errors via link_error.
• “How does Celery know when all chord body tasks finished?” — chord_unlock task polls the result backend; counts completed tasks; fires the callback when all are done. Requires a result backend that supports it (Redis preferred).
• “What’s the perf cost of a 1000-task group?” — 1000 enqueues, 1000 dequeues, 1000 result-backend writes. Group result lives in the backend until consumed. Big memory hit on Redis. For very-large fan-outs, batch (chunks=100) or use a streaming pattern.
• “When wouldn’t you use Canvas?” — complex workflows with branching, compensations, long-running steps, retries-per-step. Reach for Temporal or Step Functions instead. Canvas is best for simple linear pipelines and parallel fan-outs.
• “How do you preserve order across a group result?” — embed an index in each task’s input and output, sort in the callback. Group results aren’t guaranteed ordered by Celery.