Celery Beat: HA, Scheduling, and Lock Contention

Celery Beat is the scheduler — it enqueues tasks on a schedule. By default it’s a single process; the schedule is in a local file. If Beat dies, scheduled tasks stop firing. If you run two Beats, you get duplicate task fires.

Why single-Beat is a problem

# celery_app.py
app.conf.beat_schedule = {
    "send-daily-report": {
        "task": "tasks.send_daily_report",
        "schedule": crontab(hour=8, minute=0),
    },
    "cleanup-old-sessions": {
        "task": "tasks.cleanup_sessions",
        "schedule": 3600.0,
    },
}

celery -A celery_app beat --loglevel=info

This runs one Beat process. Single point of failure:

• Process crashes → schedule stops.
• Pod evicted in K8s → schedule stops until new pod starts.
• You deploy and the old Beat dies before the new one starts → missed window.

You can’t just run two Beats — both will enqueue the task, every cron fire produces 2 task executions.

The classic solution: lock-based singleton

Two Beat replicas; only one is “active” at a time, holding a distributed lock. If the active one dies, the other takes over after the lock TTL expires.

Option 1: celery-redbeat

redbeat is a Redis-backed scheduler that replaces the default. The schedule is in Redis (instead of a local file), and Beat uses Redis as the source of truth for “is now the right time”.

app.conf.beat_scheduler = "redbeat.RedBeatScheduler"
app.conf.redbeat_redis_url = "redis://redis:6379/1"
app.conf.redbeat_lock_timeout = 600       # 10 min; lock-holding instance must heartbeat

Run multiple Beat instances. They contend for a Redis lock; only the holder enqueues tasks. If it dies, another picks up after redbeat_lock_timeout.

Pros: clean HA, schedule is centralized, can be modified at runtime via API. Cons: extra Redis dependency, lock-timeout window means brief gaps on failover.

Option 2: celery-beatx / celery-singleton

Similar idea via different libraries. Pick one that’s actively maintained.

Option 3: Postgres advisory lock around stock Beat

If you don’t want to introduce redbeat:

# Wrap Beat startup
import psycopg

def acquire_lock():
    conn = psycopg.connect(...)
    cur = conn.cursor()
    cur.execute("SELECT pg_try_advisory_lock(%s)", (BEAT_LOCK_ID,))
    if not cur.fetchone()[0]:
        print("Another beat is running; sleeping")
        return None
    return conn   # keep alive to hold the lock

conn = acquire_lock()
if conn:
    # Run stock celery beat
    from celery.bin.beat import beat
    beat(...)

When the holding process dies, the connection closes; the advisory lock auto-releases; another instance picks it up. Simple and robust.

Option 4: K8s with replicas=1 and PDB

If you’re on K8s, just run Beat as a Deployment with replicas: 1 and a PodDisruptionBudget of minAvailable: 0 (so it can be drained), accept the brief downtime during deploys, and rely on liveness probes to restart on crash.

Simple, no extra tooling. Downside: gaps during pod restart (~30s typically).

Schedule storage

For dynamic schedules (cron expressions managed by users / admins), django-celery-beat is the standard:

app.conf.beat_scheduler = "django_celery_beat.schedulers:DatabaseScheduler"

Crontab vs interval

from celery.schedules import crontab

beat_schedule = {
    "every-30-seconds": {
        "task": "...",
        "schedule": 30.0,                                # interval in seconds
    },
    "daily-at-8am-utc": {
        "task": "...",
        "schedule": crontab(hour=8, minute=0),           # cron expression
    },
    "weekly-monday": {
        "task": "...",
        "schedule": crontab(hour=0, minute=0, day_of_week=1),
    },
}

Crontab respects timezone (app.conf.timezone). Without timezone, schedules use UTC.

Catch-up behavior

If Beat is down for 10 minutes and missed a crontab(minute=*/5) fire, what happens?

By default: the missed fires are lost. Beat doesn’t catch up. The next fire happens at the next scheduled tick.

If you need “fire every 5 minutes, no exceptions”, you must replay missed work after recovery. This is a job for a workflow engine (Temporal), not Beat.

Beat scheduling vs at-least-once

Beat firing is not idempotent. If two Beats run for 5 seconds (e.g., during a botched failover), each fires the task once → duplicate execution. Combine with task-level idempotency.

@app.task
def cleanup_sessions():
    if not r.set("cleanup:running", "1", nx=True, ex=300):
        return   # another instance already running
    try:
        Session.objects.filter(expired_at__lt=now()).delete()
    finally:
        r.delete("cleanup:running")

Drift and timing

Beat’s schedule isn’t precisely on-time. The Beat process wakes every beat_sync_every (or sleeps to next event) and enqueues; the worker picks up and runs. Cron-style precision: ±10 seconds is normal. For sub-second scheduling, you need a different tool.

Beat itself is single-threaded — heavy schedules with many tasks can introduce latency in firing. Split into multiple Beat instances per task family if needed (each with its own lock).

Common production issues

• Two Beats running. Duplicate fires. Symptom: every scheduled task runs twice. Check pod count, deployment strategy, scheduler config.
• Beat container has no persistent storage. celerybeat-schedule file is recreated on each start; first-run-after-restart fires every task immediately (no recorded “last run”). Mount the file or use redbeat / db scheduler.
• Timezone mismatch. Beat in UTC, ops expecting local time. Set app.conf.timezone explicitly.
• Beat lock timeout too short. Lock expires during a brief network blip; second Beat takes over; first one’s lock heartbeat fails; both Beats running. Set the lock TTL generously (5-10 min).
• Worker pool can’t keep up with Beat fire rate. Tasks queue, then expire, then run late. Beat fires aren’t slowing down; the worker fleet is the bottleneck.

When to skip Beat entirely

For dynamic / user-defined schedules, complex retry semantics, or sub-second precision:

• Temporal with scheduled workflows — durable, composable, retries first-class.
• AWS EventBridge Scheduler — managed cron, integrates with Lambda/SQS.
• Kubernetes CronJobs — for periodic batch jobs running their own container, no Celery needed.

Beat is fine for “run this Celery task every X minutes” within a Celery deployment. Beyond that, evaluate alternatives.

Interview angle

• “What’s Celery Beat and why is it a SPOF?” — the scheduler that enqueues tasks on a schedule. Default deployment is a single process; if it dies, scheduled tasks stop firing. Running multiple Beats naively → duplicate fires.
• “How do you make Beat HA?” — distributed lock-based singleton. Libraries: redbeat (Redis-backed). Or wrap stock Beat with a Postgres advisory lock. Or K8s Deployment replicas: 1 and accept brief downtime.
• “What does redbeat do differently?” — schedule lives in Redis (not a local file); multiple Beat instances contend for a Redis lock; only the holder fires tasks. Tolerates Beat instance loss with redbeat_lock_timeout failover window.
• “What happens if Beat is down for 30 minutes?” — missed schedules are lost; Beat doesn’t catch up. The next scheduled fire happens normally. For “must fire every time” semantics, use a workflow engine.
• “What’s django-celery-beat?” — Django-integrated scheduler with the schedule stored in the Django DB. Admin UI for managing cron entries. Good when business users / ops need to tweak schedules without code changes.
• “How do you handle duplicate fires during Beat failover?” — task-level idempotency. Beat fires aren’t idempotent across instance transitions; the task itself needs to be (DB-level lock, Redis lock, or natural idempotency).