backend / message queues / celery / 05_dead_letter_queue.md

Dead-Letter Queue (DLQ) for Celery

6 interview angles 5 min read source

Dead-Letter Queue (DLQ) for Celery

When a task exhausts retries, where does it go? Celery’s default: nowhere — it’s marked failed in the result backend and the broker drops the message. For anything load-bearing, this is unacceptable. The DLQ pattern routes terminally-failed tasks to a side queue for investigation, replay, or alerting.

Why you want a DLQ

Without one:

  • A poison-pill message (malformed payload) that fails forever gets retried until max_retries and is discarded silently.
  • A bug that fails all send_email tasks goes unnoticed until users complain.
  • After the bug is fixed, you can’t re-run the failed work.

With one:

  • Failed messages are preserved on a dedicated queue.
  • Alerting on DLQ depth surfaces problems quickly.
  • After fixing the bug, replay the DLQ.

Celery doesn’t have a built-in DLQ

You build it. Three approaches depending on your broker.

Approach 1: Broker-native DLQ (RabbitMQ)

RabbitMQ has Dead-Letter Exchanges (DLX) — set a queue’s x-dead-letter-exchange, and rejected/expired messages get routed there automatically.

# Celery configuration
app.conf.task_queues = (
    Queue(
        "default",
        Exchange("default"),
        routing_key="default",
        queue_arguments={
            "x-dead-letter-exchange": "dead-letter",
            "x-dead-letter-routing-key": "default.dlq",
        },
    ),
    Queue(
        "default.dlq",
        Exchange("dead-letter"),
        routing_key="default.dlq",
    ),
)

When a task rejects a message (no retry left), RabbitMQ routes it to the DLQ. Configure Celery to reject on exhaustion:

@app.task(bind=True, max_retries=3, autoretry_for=(SMTPError,))
def send_email(self, email):
    try:
        smtp.send(email)
    except SMTPError as exc:
        # Celery retries up to max_retries
        raise self.retry(exc=exc) from exc
    except Exception:
        # Non-retryable; goes to DLQ via reject
        raise

Celery normally just acks the task as failed without rejecting. To get DLQ routing, you need a custom strategy or to use task_reject_on_worker_lost = True for crash cases.

Approach 2: Broker-native DLQ (SQS)

SQS has first-class DLQ support — set maxReceiveCount and redrivePolicy:

aws sqs create-queue --queue-name my-tasks
aws sqs create-queue --queue-name my-tasks-dlq

aws sqs set-queue-attributes \
  --queue-url $TASKS_URL \
  --attributes RedrivePolicy='{"deadLetterTargetArn":"arn:aws:sqs:...:my-tasks-dlq","maxReceiveCount":"5"}'

A message that’s received and not deleted 5 times moves to my-tasks-dlq. Note: SQS DLQ counts deliveries, not Celery retries. A Celery worker that picks up a message, retries it 5 times via self.retry() (which republishes via the broker), looks like 5 deliveries to SQS — the original eventually goes to DLQ.

Setting maxReceiveCount to a generous value (e.g., 10-15) handles this if you use Celery retries.

Approach 3: Application-level DLQ

When you don’t trust broker semantics, route to a side queue in code:

@app.task(bind=True, max_retries=3, autoretry_for=(TempError,))
def process_order(self, order_id):
    try:
        do_work(order_id)
    except Exception as exc:
        if self.request.retries >= self.max_retries:
            # Send to DLQ task
            dlq_handler.delay({
                "task": self.name,
                "args": self.request.args,
                "exception": str(exc),
                "traceback": traceback.format_exc(),
                "id": self.request.id,
            })
            return        # don't raise; ack as done
        raise self.retry(exc=exc)

@app.task
def dlq_handler(payload):
    DeadLetter.objects.create(**payload)
    sentry.capture_message("Task in DLQ", extra=payload)

Now failed tasks are rows in your DB; replay them with a management command.

Replaying the DLQ

Once the bug is fixed, drain the DLQ back to the original queue.

# RabbitMQ DLQ → main queue
@app.task
def replay_dlq():
    with rabbitmq.connect() as conn:
        with conn.channel() as ch:
            while True:
                method, props, body = ch.basic_get("my-tasks.dlq", auto_ack=False)
                if not method:
                    break
                ch.basic_publish(exchange="", routing_key="my-tasks", body=body, properties=props)
                ch.basic_ack(method.delivery_tag)

For SQS: AWS Console has a “Redrive” button or use the CLI:

aws sqs start-message-move-task \
  --source-arn $DLQ_ARN \
  --destination-arn $TASKS_QUEUE_ARN

For application-level DLQ, a management command re-queues from your DB table:

def handle(self, *args, **options):
    for row in DeadLetter.objects.filter(status="pending"):
        task = app.tasks[row.task]
        task.delay(*row.args)
        row.status = "replayed"
        row.save()

Alerting

# Track DLQ depth as a CloudWatch metric
@periodic_task(run_every=60)
def report_dlq_metrics():
    count = DeadLetter.objects.filter(status="pending").count()
    cloudwatch.put_metric_data(
        Namespace="MyApp/Celery",
        MetricData=[{"MetricName": "DLQDepth", "Value": count, "Unit": "Count"}],
    )

Alert on:

  • DLQ depth > 0 (any DLQ activity is worth a look).
  • DLQ depth growing.
  • DLQ depth > N for > T minutes.

What to put in the DLQ envelope

{
  "task_name": "send_welcome_email",
  "args": [...],
  "kwargs": {...},
  "exception": "SMTPException(...)",
  "traceback": "...",
  "first_seen": "2024-01-15T10:30:00Z",
  "last_attempt": "2024-01-15T11:00:00Z",
  "attempt_count": 5,
  "task_id": "uuid",
  "trace_id": "...",
  "user_id": 42
}

The traceback + trace_id let you triage. The args/kwargs let you replay. The attempt_count and last_attempt let you tell “still failing” from “old failure, fixed now”.

Anti-patterns

  • No DLQ at all. Silent failure. Discovered by user complaint.
  • DLQ depth not alerted. Items pile up; bugs go uninvestigated.
  • Replaying DLQ without fixing the bug. Same messages bounce back instantly.
  • DLQ items in the broker forever. Storage cost; cleanup policy needed.
  • Storing PII / secrets in the DLQ. DLQ entries persist long-term; tighten access or scrub sensitive fields.

Tools

  • Flower — Celery’s web UI; shows failed tasks but doesn’t have a real DLQ feature.
  • Sentry — task failure reporting via celery-sentry. Good for tracking errors; not a DLQ.
  • AWS SQS DLQ console — UI to inspect, redrive.
  • RabbitMQ Management UI — inspect queues, requeue messages manually.

Interview angle

  • “What happens when a Celery task exhausts retries?” — by default, marked failed in the result backend and silently dropped from the queue. No alert, no recoverable state. You need to build a DLQ pattern explicitly.
  • “How do you build a DLQ for Celery?” — three approaches: (a) RabbitMQ DLX (set x-dead-letter-exchange on the queue); (b) SQS native DLQ (redrivePolicy + maxReceiveCount); (c) application-level — on retry exhaustion, route to a dedicated DLQ task that writes a DB row.
  • “What goes in the DLQ message?” — task name, original args/kwargs, the exception + traceback, first/last attempt timestamps, attempt count, trace ID. Enough to triage and replay.
  • “How do you alert on DLQ depth?” — periodic task counts the DLQ size, emits a CloudWatch / Prometheus metric, alarm on threshold. Any DLQ activity is worth investigating; sustained growth is a fire.
  • “How do you replay the DLQ?” — after fixing the bug: RabbitMQ → use shovel or a replay task to move DLQ → main queue. SQS → AWS Console “Redrive” or StartMessageMoveTask. App-level → management command iterates rows and re-enqueues.
  • “What’s a poison message and how do DLQs help?” — a message that fails every time (malformed payload, missing reference). Without DLQ, it blocks the queue (other messages backed up behind it on FIFO brokers) or wastes worker time. DLQ pulls it out of the main flow; alerting surfaces the bug.
Previous
Celery Task Idempotency