RQ (Redis Queue)

The simple alternative to Celery. Worth knowing mainly so you can argue for it when Celery is overkill — which is often.

The model

Jobs are Python callables serialised onto a Redis list. Workers pop and execute.

from redis import Redis
from rq import Queue

q = Queue("default", connection=Redis())
job = q.enqueue(send_email, user_id=42, template="welcome")

job.id, job.get_status()          # "queued" -> "started" -> "finished" / "failed"
job.result                        # available once finished

rq worker default high low        # priority order: drains high first

That’s essentially the whole API. The simplicity is the point.

RQ vs Celery

RQ forks a process per job. That gives real isolation — a segfault or memory leak in one job can’t affect the worker — at the cost of fork overhead per job and no Windows support.

Celery’s advantages are genuine when you need them: multiple broker backends, complex workflow composition, mature scheduling, and fine-grained concurrency models. Most applications need none of that.

When RQ is the right answer

• You already run Redis and don’t want another broker.
• Jobs are independent — no chains, chords or fan-in.
• The team is small and Celery’s configuration surface is a liability rather than an asset.
• You want to read the source when something misbehaves. RQ is small enough to do that.

When Celery wins: RabbitMQ or SQS as the broker, complex workflow orchestration, high throughput where per-job fork cost matters, or an existing Celery investment.

Neither is the answer for durable multi-step business processes — that’s Temporal, which gives you retries, compensation and multi-day durability as first-class concepts rather than things you assemble. See ../temporal/03_temporal_vs_celery.md.

Reliability

from rq import Retry

q.enqueue(flaky_call, retry=Retry(max=3, interval=[10, 60, 300]))

The failure modes to understand:

• Redis is not a durable broker by default. With the default persistence settings, a Redis crash can lose queued jobs. Enable AOF with appendfsync everysec if job loss matters — and accept that “everysec” means up to a second of loss.
• At-least-once, not exactly-once. A worker killed mid-job leaves the job in the started registry. Make jobs idempotent; this is not optional. See ../celery/04_idempotency.md.
• Failed jobs land in the FailedJobRegistry rather than vanishing — inspect and requeue from there.
• Set job_timeout. A job with no timeout can hold a worker forever.

q.enqueue(process_file, path, job_timeout="10m", result_ttl=3600)

result_ttl matters at volume: results persist in Redis, and forgetting to expire them is a slow memory leak.

Practical notes

• Arguments are pickled. Pass IDs, not ORM objects — pickling a SQLAlchemy instance serialises a detached object with a stale session. Pass user_id, load inside the job.
• rq-scheduler or enqueue_at for delayed and periodic work. Less capable than Celery Beat; for anything complex, use a real scheduler.
• Monitoring: rq-dashboard, or scrape queue depth into Prometheus. Queue depth and oldest-job-age are the two alerts worth having.
• Deploys: send SIGTERM for a warm shutdown so the worker finishes its current job before exiting.

Interview angle

• “RQ or Celery?” — RQ when you already run Redis, jobs are independent, and you value a small readable codebase. Celery when you need a different broker, complex workflow composition, or mature scheduling. Most applications don’t need Celery’s surface area.
• “How does RQ execute jobs?” — it forks a process per job, which isolates crashes and leaks from the worker, at the cost of fork overhead and no Windows support.
• “Is Redis a safe broker for jobs?” — not by default. Default persistence can lose queued jobs on a crash. Enable AOF if that matters, and accept the sub-second loss window. Delivery is at-least-once regardless, so jobs must be idempotent.
• “Why pass IDs rather than objects to a task?” — arguments are pickled. An ORM instance serialises detached from its session, so the worker gets stale data or an error. Pass the ID and load inside the job.
• “When is neither RQ nor Celery right?” — durable multi-step business processes with compensation and multi-day timelines. That’s a workflow engine like Temporal, where retries, sagas and durability are first-class rather than assembled from task primitives.