Practical Case: Webhook Ingestion & Processing Service

Scenario

A third party (Stripe, GitHub, a partner API) sends you webhooks. You must receive them, verify they’re genuine, process them reliably, and never lose or double-apply one. The provider retries on any non-2xx, so your endpoint has to be fast and idempotent.

This is a classic senior take-home. The naive version — “parse the JSON and do the work in the handler” — fails on every axis that matters: it’s slow (provider times out and retries), it loses events (worker crashes mid-processing), and it double-applies (retry of an event you already handled). The interesting design is the receive/process split.

Stack: FastAPI + Postgres + Celery + Redis on AWS.

How would you design this?

Split it into two phases with a durable boundary between them.

1. Receive (synchronous, <50ms) — verify the signature, persist the raw event to Postgres, return 200 immediately. Do no business logic here.
2. Process (asynchronous) — a Celery worker picks up the event, runs the business logic, marks it done. Retries and failures happen here, invisible to the provider.

[provider] --POST--> [FastAPI: verify sig, INSERT raw event, 200]
                                   |
                                   v  (enqueue event id)
                              [Redis broker]
                                   |
                                   v
                          [Celery worker: process, mark done]
                                   |
                          [Postgres business tables]

The durable boundary is the webhook_events table. Once the row is committed, the event is safe — even if every worker is down, the event is not lost; it’s processed when workers come back.

The receive endpoint

@app.post("/webhooks/stripe")
async def receive(request: Request):
    body = await request.body()
    sig = request.headers.get("Stripe-Signature", "")

    if not verify_signature(body, sig, WEBHOOK_SECRET):
        raise HTTPException(400, "bad signature")

    event = json.loads(body)
    try:
        await db.execute(
            insert(webhook_events).values(
                id=event["id"],          # provider's event id = idempotency key
                type=event["type"],
                payload=body,
                status="pending",
            )
        )
    except UniqueViolation:
        return {"status": "duplicate"}   # already received — provider retry, ignore

    celery_app.send_task("process_webhook", args=[event["id"]])
    return {"status": "accepted"}

Key points:

• Verify the signature on the raw bytes, before parsing. Re-serialized JSON won’t match the HMAC. This is the #1 webhook bug.
• The provider’s event id is the idempotency key. A unique constraint on id makes duplicate receives a no-op — the provider retrying a webhook you already stored just hits the constraint.
• Persist before enqueue. If you enqueue first and the DB write fails, you have a task referencing a row that doesn’t exist. DB-write-then-enqueue means the worst case is a committed row that was never enqueued — recoverable by a sweeper (below).
• Return 2xx fast. Any slow work here = provider timeout = provider retry = load amplification.

The processing worker

@celery_app.task(
    name="process_webhook",
    bind=True,
    max_retries=5,
    retry_backoff=True,        # exponential: 1s, 2s, 4s, 8s, 16s
    retry_backoff_max=600,
    retry_jitter=True,
)
def process_webhook(self, event_id: str):
    with db.begin():           # one transaction
        event = db.query(WebhookEvent).filter_by(id=event_id) \
                  .with_for_update().one()

        if event.status == "done":
            return             # already processed — idempotent

        handler = HANDLERS[event.type]
        handler(json.loads(event.payload))   # the business logic

        event.status = "done"
        event.processed_at = func.now()

Key points:

• SELECT ... FOR UPDATE locks the row so two workers can’t process the same event concurrently (e.g. a duplicate enqueue, or a retry overlapping the original).
• Check status == "done" inside the lock. This is processing idempotency — distinct from the receive idempotency the unique constraint gave you.
• Business logic + status update in one transaction. If the handler half-succeeds and the process crashes, the transaction rolls back and the event stays pending — it gets retried cleanly. Never commit the business change and the status flip separately.
• At-least-once, made safe by idempotency. Celery (like SQS) is at-least-once. You don’t fight that; you make the handler safe to run twice. If the handler itself isn’t naturally idempotent (e.g. “send an email”), give it an idempotency key too.
• max_retries then dead-letter. After N failed retries, the task lands in a dead-letter state — set status="failed", alert, and move on. One poison event must not block the rest.

What about events that never get processed?

The persist-then-enqueue gap (DB committed, enqueue failed; or the broker dropped the message) leaves pending rows that no worker knows about. A sweeper closes the gap:

@celery_app.task
def sweep_stuck_events():
    stuck = db.query(WebhookEvent).filter(
        WebhookEvent.status == "pending",
        WebhookEvent.created_at < utcnow() - timedelta(minutes=5),
    ).limit(100)
    for event in stuck:
        celery_app.send_task("process_webhook", args=[event.id])

Run it every minute via Celery Beat. It re-enqueues anything stuck — and because processing is idempotent, re-enqueuing something that is actually in flight is harmless. This turns “exactly-once delivery” (impossible) into “at-least-once delivery + idempotent processing” (achievable).

On AWS — what runs where

A senior variant: drop Celery+Redis, use SQS directly. API Gateway → Lambda (verify + persist + SendMessage) → SQS → Lambda consumer, with an SQS DLQ. Fewer moving parts, native at-least-once + DLQ, no broker to operate. Celery wins if you need its scheduling/chaining or you’re already invested in it.

What can go wrong

• Signature verified on parsed JSON — re-serialization changes bytes, HMAC fails. Verify raw bytes.
• Business logic in the receive handler — slow response, provider retries, you’ve built a self-amplifying load problem.
• No idempotency — provider retries (and they will retry) double-apply. Unique constraint on receive, status check on process.
• Enqueue before persist — task references a missing row. Persist first.
• Committing business change and status flip separately — crash between them = double-apply or stuck. One transaction.
• Out-of-order delivery — webhooks are not ordered. A subscription.updated can arrive before subscription.created. Use the payload’s own timestamps/versioning, don’t assume arrival order.
• One poison event blocks the queue — bound retries, dead-letter, alert, keep moving.
• Replay attacks — an attacker resends a captured (validly-signed) webhook. Reject events whose timestamp is too old; the idempotency key also limits the blast radius.

Interview angle

1. “The provider says they delivered an event but you have no record of it — what happened?” — Likely your endpoint returned non-2xx (or timed out) and you didn’t persist; or you persisted but a deploy dropped in-flight requests. Check: are you returning 2xx after the DB commit? Is the receive path actually fast?
2. “How do you guarantee exactly-once processing?” — You don’t — delivery is at-least-once. You get effectively exactly-once by making processing idempotent: dedup on receive (unique constraint), dedup on process (status check under a row lock), business logic + status in one transaction.
3. “Why not just process synchronously in the handler?” — The provider has a short timeout; real work blows it, triggering retries and load amplification. No async means no retry isolation — a transient downstream failure becomes a lost event. The receive/process split gives you a fast ack and a durable, retryable backlog.
4. “A bug corrupted how you processed the last 1000 events — how do you reprocess?” — You kept the raw payloads in webhook_events. Reset those rows to pending (or a reprocess status) and let the sweeper re-enqueue them. Storing the raw event is what makes backfills possible.
5. “Webhooks arrive out of order — updated before created. How do you handle it?” — Don’t rely on arrival order. Use the version/sequence number or timestamp inside the payload; if you get an updated for an entity you don’t have, either fetch current state from the provider’s API or hold it until the created arrives.

Cross-links:

• Idempotent processing in depth: ../system_design/07_worked_designs/08_idempotent_payments.md
• Job scheduler / FOR UPDATE SKIP LOCKED: ../system_design/07_worked_designs/07_job_scheduler.md
• Celery vs asyncio for background work: ../system_design/03_async_patterns/01_async_io_and_background_work.md
• Resilience (retries, backoff, DLQ): ../system_design/02_resilience/