Celery — Common Interview Questions and Answers

1. What is Celery and when should you use it?

Celery is a distributed task queue for Python. It runs tasks asynchronously in worker processes, using a message broker (e.g. Redis, RabbitMQ) to pass messages.

Use it when you need:

• Background jobs (emails, reports, file processing)
• Periodic/scheduled tasks (Celery Beat)
• Offloading heavy or long-running work from the web process
• Distributing work across many workers or machines

2. What are the main components of Celery?

• Application/Client: Your code that defines and sends tasks (e.g. task.delay()).
• Broker: Message queue (Redis, RabbitMQ, etc.) that stores task messages.
• Workers: Processes that pull messages from the broker and run tasks.
• Result backend (optional): Stores task results (Redis, DB, etc.).
• Celery Beat: Scheduler that sends periodic tasks to the broker.

Flow: Client → Broker → Worker → (optional) Result backend.

3. How do you define and call a Celery task?

from celery import Celery

app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task
def add(x, y):
    return x + y

# Asynchronous call (returns immediately)
result = add.delay(4, 5)

# Get result (blocks until done)
print(result.get(timeout=10))  # 9

.delay(*args, **kwargs) is shorthand for .apply_async(args, kwargs).

4. What is the difference between Redis and RabbitMQ as a Celery broker?

RabbitMQ is often preferred in production for reliability and features; Redis is common for development and simpler deployments.

5. How do you retry a task on failure?

Use bind=True so the task receives self, then call self.retry() in an exception handler:

@app.task(bind=True, max_retries=3)
def fetch_url(self, url):
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.RequestException as exc:
        raise self.retry(exc=exc, countdown=60)

countdown is seconds before the next try; you can implement exponential backoff with countdown=60 * (2 ** self.request.retries).

6. What are Celery task states and how do you check them?

Common states: PENDING, STARTED, SUCCESS, FAILURE, RETRY, REVOKED.

result = add.delay(4, 5)
result.state   # 'PENDING', then 'SUCCESS', etc.
result.ready() # True when finished (success or failure)
result.successful()  # True only on success
result.get()   # Blocks and returns return value or raises

You need a result backend to get reliable state and result after the worker finishes.

7. What is Celery Beat and how do you use it?

Celery Beat is a scheduler that sends task messages at fixed intervals (like cron). You define a schedule and run the beat process:

from celery.schedules import crontab

app.conf.beat_schedule = {
    "every-midnight": {
        "task": "tasks.cleanup",
        "schedule": crontab(hour=0, minute=0),
    },
    "every-5-minutes": {
        "task": "tasks.sync",
        "schedule": 300.0,  # seconds
    },
}

Run: celery -A proj beat (and workers with celery -A proj worker).

8. How do you use different queues and route tasks to them?

Define queues and route tasks via CELERY_TASK_ROUTES or the task’s queue:

app.conf.task_queues = {
    "high": {"exchange": "high", "routing_key": "high"},
    "low": {"exchange": "low", "routing_key": "low"},
}
app.conf.task_routes = {
    "tasks.send_email": {"queue": "high"},
    "tasks.cleanup": {"queue": "low"},
}

Start workers for a queue: celery -A proj worker -Q high,default.

9. What is task idempotency and why does it matter?

Idempotency means running the same task more than once (e.g. retries, duplicate messages) has the same effect as running it once. Design tasks so that:

• Duplicate execution doesn’t double-charge, double-send emails, or corrupt data.
• Use unique keys (e.g. job id, payment id) and check “already processed” when needed.

10. How do you limit concurrency or rate of tasks?

• Concurrency: Start workers with limited concurrency, e.g. celery -A proj worker --concurrency=2.
• Rate limits: Use rate_limit on the task:

@app.task(rate_limit="10/m")  # 10 per minute
def api_call():
    ...

• Global rate: Configure broker/worker or use a dedicated rate-limiting mechanism for external APIs.

11. What is the difference between delay() and apply_async()?

• task.delay(a, b, c) — simple call with positional (and keyword) args; default options.
• task.apply_async(args, kwargs, countdown=60, queue="high", ...) — full control over ETA, queue, retries, etc.

Use apply_async when you need countdown, ETA, queue, or other options.

12. How do you revoke or cancel a task?

Revoke by task id (worker will skip or terminate the task depending on configuration):

from proj.celery import app

result = my_task.delay()
result.revoke()  # or
app.control.revoke(result.id, terminate=True)

terminate=True tries to kill the worker process running the task; use with care.

13. How do you chain or group tasks?

• Chain: run tasks in sequence, each receiving the previous result: chain(task_a.s(1), task_b.s(), task_c.s()).apply_async()
• Group: run tasks in parallel: group(task_a.s(1), task_a.s(2)).apply_async()
• Chord: group + callback when all finish: chord(group(...), callback.s()).apply_async()

All return AsyncResult-like objects; you need a result backend for chord callback and result retrieval.

14. What are common pitfalls when using Celery?

• Serialization: Don’t pass non-serializable objects (DB models, file handles); pass ids or simple data.
• No result backend: Without it, result.get() and state are not reliable after the worker finishes.
• Long-running tasks: Risk of worker killed by broker timeouts; consider chunking or heartbeats.
• Visibility timeout (SQS): Message can be redelivered if the task runs longer than the visibility timeout.
• Same broker for results and broker: Possible under memory pressure; separate Redis DB or use different stores.

15. How do you run Celery in production?

• Use a process manager (systemd, supervisord) or container orchestration to run celery worker and celery beat.
• Prefer RabbitMQ (or a managed queue) for the broker in production.
• Tune worker concurrency (e.g. --concurrency=4) and use enough workers for throughput.
• Use a result backend if you need task results or status.
• Set timeouts and retry limits; monitor queues and worker health.
• Run beat as a single instance to avoid duplicate scheduled tasks.

Interview angle

• “Redis or RabbitMQ as the broker?” - RabbitMQ is a real message broker with routing, confirms and durability guarantees; Redis is simpler and faster but can lose messages depending on persistence settings. Redis is fine for best-effort work, RabbitMQ for anything that must not be lost.
• “What are the canvas primitives?” - chain for sequential, group for parallel, chord for a group plus a callback that runs when all finish. Chords need a result backend and are the most fragile part of Celery; for anything complex a workflow engine is a better fit.
• “How do you schedule periodic tasks?” - Celery Beat. Note the single-scheduler constraint: running two Beat instances double-fires everything, so it needs a lock or a single-replica deployment.
• “When would you not use Celery?” - long multi-step business processes with compensation, where Temporal’s durable execution is a better fit, or simple in-process work where a background task suffices. See ../temporal/03_temporal_vs_celery.md.