Worked Design — Distributed Job Scheduler

“Design a system that runs scheduled and background jobs” (cron-as-a-service / a task queue platform). Tests scheduling, exactly-once-ish execution, retries, and the worker pool. Stack: FastAPI + Postgres + Redis/SQS + Celery-style workers.

1. Requirements

Functional: submit a job to run now (background task), later (at a timestamp), or on a schedule (cron); workers execute jobs; jobs that fail retry; visibility into job status.

Non-functional: a scheduled job must run at most once per trigger (no duplicate runs from a double-fire) and ideally at least once (don’t silently skip a run); durable (a submitted job survives a crash); scalable workers; no single point of failure in the scheduler itself.

Clarify: is “exactly once” required, or “at least once + idempotent jobs”? Max job duration? Priorities? How precise must scheduled times be (±1s? ±1min?)?

2. Scale

1M scheduled jobs, ~100k job executions/hour at peak → ~30/sec, bursty (many cron jobs fire at :00)
job durations: milliseconds to minutes

The bursty pattern matters: cron jobs cluster on round times (every job set to “hourly” fires at the same instant). The system must absorb that spike.

3. The three job types

The unifying idea: everything becomes a row with a run_at; the scheduler/poller’s job is to move rows whose time has come into the ready queue, and workers consume the ready queue.

4. Architecture

API ─► Postgres (jobs table: definition, run_at, status)
                 │
        ┌────────┴─────────┐
        ▼                  ▼
   Scheduler           (recurring jobs: compute next run_at, insert next execution)
   (poller)
        │  "SELECT ... WHERE run_at <= now() AND status='pending'
        │   FOR UPDATE SKIP LOCKED LIMIT 100"
        ▼
   Ready Queue (Redis / SQS)
        │
        ▼
   Worker pool ──► execute ──► update status, handle retry/DLQ

• Jobs table in Postgres — the durable source of truth. Each job: definition (what to run), run_at, status (pending/running/succeeded/failed), attempt count, cron expression if recurring.
• Scheduler/poller — periodically queries for jobs whose run_at has passed and pushes them to the ready queue. The key SQL trick: FOR UPDATE SKIP LOCKED lets multiple scheduler instances poll concurrently without handing the same job to two of them.
• Ready queue — decouples “this job is due” from “a worker is free.” Absorbs the :00 burst.
• Worker pool — pulls from the ready queue, executes, updates status. Autoscales on queue depth.

5. The hard part — the scheduler must be HA without double-firing

A single scheduler instance is a SPOF (it dies, nothing gets scheduled). Run several — but then they could each fire the same job. Two ways to make the scheduler HA and exactly-once-per-trigger:

Option A — leader election. Only one scheduler instance is active at a time; the others stand by. Leadership via a distributed lock (Postgres advisory lock, Redis lock, or etcd/ZooKeeper lease). Leader dies → a standby acquires the lock and takes over. Simple; the leader is a throughput ceiling but scheduling is light work.

Option B — partitioned polling with SKIP LOCKED. All scheduler instances poll, but SELECT ... FOR UPDATE SKIP LOCKED guarantees each pending job row is claimed by exactly one instance. No leader, naturally scales, no failover gap. This is the cleaner answer for Postgres-backed systems and the one to lead with.

Either way, the principle: the claim must be atomic. A job goes from pending to running (or onto the queue) in one atomic step, so no two schedulers and no two workers can both grab it.

6. Exactly-once is a lie — design for at-least-once + idempotency

True exactly-once execution across a distributed system with crashes is not achievable. What you actually build:

• At-least-once delivery — the job will run; on a worker crash mid-job, it’ll be retried (the row goes back to pending after a visibility timeout, or a reaper requeues stuck running jobs).
• Idempotent jobs — the job itself must be safe to run twice (dedup key, INSERT ... ON CONFLICT, check-then-act guarded by a unique constraint). This is the job author’s responsibility, and the platform should make it easy (pass each execution a stable execution_id the job can dedup on).

So the honest answer to “exactly once?” is: “at-least-once execution + idempotent jobs = exactly-once effect.”

7. Retries, timeouts, failure handling

• Visibility timeout / lease — when a worker claims a job, it gets a lease (or the SQS visibility timeout). If the worker crashes, the lease expires and the job becomes claimable again. Long jobs must heartbeat to extend the lease, or they get re-run while still running.
• Retry with exponential backoff + jitter on failure, up to max_attempts.
• Dead-letter — after max_attempts, the job goes to a failed/DLQ state; alert on it; it’s a bug or a poison job.
• Stuck-job reaper — a periodic sweep that finds jobs running past a sane max duration with an expired lease and requeues or fails them — covers workers that died without releasing the lease.

8. Missed runs & catch-up

If the scheduler is down for 10 minutes, scheduled jobs in that window didn’t fire. On recovery:

• Catch up — run the missed executions (good for “process this batch” jobs).
• Skip — only run the next scheduled occurrence (good for “send the daily digest” — running 10 stale digests is wrong).

Make this per-job policy, not a global behavior. The job definition declares whether missed runs catch up or are skipped.

9. Bottlenecks & trade-offs

• The :00 thundering herd — every “hourly” job fires at once. The ready queue absorbs it; workers drain it at their sustainable rate; scheduled-time precision degrades slightly during the burst (jobs run a few seconds late). Acceptable for most jobs; jitter the schedules if precision matters.
• Polling latency vs load — the poller runs every N seconds; smaller N = tighter scheduling precision but more DB load. ~1-5s polling is a typical balance; for sub-second precision you’d need a different mechanism (a timer wheel / delayed queue).
• Postgres as the queue — FOR UPDATE SKIP LOCKED makes Postgres a perfectly good job queue at moderate scale and keeps everything in one transactional store. At very high throughput, move the ready queue to Redis/SQS while keeping Postgres as the durable definition store.
• Long jobs — block a worker for minutes; size the pool accordingly, separate long-job and short-job queues so a slow job doesn’t starve quick ones, and require heartbeating so the lease doesn’t expire mid-run.
• Recurring-job drift — compute the next run_at from the schedule, not from “now + interval” after the run, or slow jobs cause the schedule to drift later and later.

Interview angle

• “How do you make the scheduler highly available without firing jobs twice?” — either leader election (one active scheduler, standbys take over via a distributed lock) or — cleaner for a Postgres-backed system — every instance polls with SELECT ... FOR UPDATE SKIP LOCKED, which atomically hands each due job to exactly one instance. The claim (pending → running/queued) must be one atomic step.
• “Can you guarantee exactly-once execution?” — no — not across a distributed system with crashes. You build at-least-once execution (crashed jobs are retried) plus idempotent jobs (each execution gets a stable id to dedup on). At-least-once + idempotency = exactly-once effect.
• “How do the three job types (now / later / cron) unify?” — everything becomes a row with a run_at. The scheduler moves rows whose time has come into a ready queue; workers consume the queue. Recurring jobs additionally compute and insert their next execution each time they fire.
• “A worker crashes mid-job — what happens?” — its lease (or SQS visibility timeout) expires and the job becomes claimable again; a stuck-job reaper also sweeps for running jobs past their max duration. The job re-runs — which is why jobs must be idempotent. Long jobs heartbeat to extend the lease so they aren’t re-run while still alive.
• “Every hourly job fires at :00 — how do you handle the burst?” — the ready queue absorbs the spike; workers drain it at a sustainable rate; jobs run a few seconds late during the burst. If tighter precision is needed, jitter the schedules so they don’t all align on the round minute.
• “The scheduler was down for 10 minutes — do missed jobs run?” — per-job policy: ‘catch up’ (run each missed occurrence — right for batch processing) or ‘skip to next’ (right for a daily digest — running 10 stale digests is wrong). The job definition declares which.
• “Why is Postgres a viable job queue here?” — FOR UPDATE SKIP LOCKED gives atomic, concurrent claiming, and keeping definitions + state in one transactional store is simple and reliable at moderate scale. At high throughput you split: Postgres stays the durable definition store, the ready queue moves to Redis/SQS.