Worked Design — Notification / Fan-out Service

“Design a system that sends notifications” — email, push, SMS, in-app. Tests async architecture, the fan-out pattern, idempotency, and third-party-failure handling. Stack: FastAPI + Postgres + Celery/SQS + Redis + SNS/SES.

1. Requirements

Functional: other services trigger notifications (“order shipped”, “new follower”); deliver across channels (email, push, SMS, in-app); respect user preferences (channel opt-outs, quiet hours); don’t send duplicates.

Non-functional: the trigger call must be fast and non-blocking (the ordering service shouldn’t wait on an email); delivery is async and best-effort with retries; at-least-once with dedup (better a retried send than a lost one, but never a duplicate the user sees); handle third-party provider outages gracefully.

Scope: templating and localization exist but aren’t the focus; analytics (open/click) is a separate concern.

2. Scale

10M users, ~5 notifications/user/day → 50M/day → ~600/sec avg, ~1,500 peak
spiky: a marketing blast or a viral event → 100k+ in a burst

The spikiness is the design driver: the system must absorb bursts without dropping work and without overwhelming downstream providers.

3. API

POST /notifications   { user_id, type, payload, idempotency_key }   → 202 Accepted
GET  /notifications?user_id&cursor                                  → in-app feed
PUT  /preferences/{user_id}    { channel_opt_outs, quiet_hours }

The trigger returns 202 immediately — it accepted the request, delivery happens asynchronously. The caller never waits.

4. Architecture — the fan-out

Service ─► POST /notifications ─► API ─► [enqueue] ─► Notification Queue
                                          │ (write to DB: status=pending)
                                          ▼
                                   Dispatcher worker
                                          │  loads user preferences
                                          │  fans out per enabled channel
                          ┌───────────────┼────────────────┐
                          ▼               ▼                ▼
                    Email Queue      Push Queue       SMS Queue
                          │               │                │
                    Email worker    Push worker      SMS worker
                          ▼               ▼                ▼
                      SES            FCM/APNs          Twilio

Why this shape:

Queue right after the API — absorbs bursts; the API stays fast and never blocks on delivery.
Dispatcher — one notification event fans out into per-channel jobs based on the user’s preferences. This is the fan-out: 1 event → up to N channel sends.
Per-channel queues + workers — each channel scales independently and fails independently. If Twilio is down, the SMS queue backs up and retries; email and push are unaffected. One slow/broken provider can’t stall the others.
DB tracks state — each notification and each channel-send has a status (pending → sent / failed / suppressed) for observability, the in-app feed, and retry bookkeeping.

5. Idempotency & dedup — the part interviewers probe

At-least-once delivery (queues redeliver, workers retry) means the same notification can be processed twice. The user must never see a duplicate. Two layers:

Producer-side idempotency key — the caller sends idempotency_key; the API does INSERT ... ON CONFLICT (idempotency_key) DO NOTHING. A retried trigger (“order shipped” fired twice by the ordering service) creates one notification, not two.

Consumer-side dedup — before a channel worker actually sends, check a dedup record (notification_id, channel); insert it in the same transaction as marking the send complete. A redelivered job hits the unique constraint and skips. For providers that support it (SES, Twilio, Stripe-style APIs), also pass a provider-side idempotency key as belt-and-suspenders.

Net effect: at-least-once delivery + idempotent consumers = exactly-once as the user experiences it.

6. Preferences & quiet hours

The dispatcher consults preferences before fan-out:

Channel opt-outs — skip disabled channels; mark those sends suppressed.
Quiet hours — if it’s 3 AM in the user’s timezone and the notification isn’t urgent, schedule it for later (a delay queue / visible_after timestamp) rather than dropping it.
Suppression list — hard bounces and complaints (from SES) feed back into a suppression store; never email a suppressed address again.

7. Retries, DLQ, provider failure

Retries with exponential backoff + jitter on transient provider errors (5xx, timeouts, rate limits — respect Retry-After). Don’t retry hard failures (invalid number, unsubscribed).
DLQ — after N attempts, the channel-send goes to a dead-letter queue. Alert on DLQ depth; it surfaces a broken provider or a poison payload.
Circuit breaker per provider — if Twilio is failing every call, stop hammering it: open the breaker, let SMS jobs sit in the queue, retry on a probe. Protects your worker pool and Twilio’s recovery.
Provider failover — for critical channels, a secondary provider (SES → a backup ESP) the worker fails over to when the primary’s breaker is open.

8. Bottlenecks & trade-offs

Burst absorption vs latency — the queue means a marketing blast doesn’t drop work, but during the burst, delivery latency rises (jobs wait in line). That’s the correct trade for notifications — delivered late beats dropped. For genuinely urgent notifications, a separate high-priority queue.
Fan-out amplification — 1 event → N channel jobs; a campaign to 1M users is 1M+ jobs. The per-channel queues + horizontally-scaled workers handle it; autoscale workers on queue depth.
Third-party rate limits — SES/Twilio/FCM all throttle. Worker concurrency must be capped to stay under the provider’s limit, or you trade your 429s for theirs.
Ordering — generally not required (“you have 3 notifications” doesn’t care about order). Don’t pay for FIFO queues unless a specific case needs it.
In-app channel is different — it’s just a write to the notifications table + a read endpoint (or a WebSocket push); no third-party, no provider failure mode.

Interview angle

“Why put a queue right after the API?” — to absorb bursts and keep the trigger call fast. The calling service gets a 202 immediately and never blocks on email/SMS delivery; the queue smooths a 100k-notification spike into a sustainable worker rate.
“What’s the fan-out and why per-channel queues?” — one notification event fans out into per-channel send jobs based on user preferences. Separate queues + workers per channel so each scales independently and fails independently — a Twilio outage backs up SMS without touching email or push.
“How do you guarantee no duplicate notifications under at-least-once delivery?” — two layers: producer-side idempotency key (INSERT ... ON CONFLICT DO NOTHING on the trigger) so a double-fired event creates one notification; consumer-side dedup record (notification_id, channel) inserted in the same transaction as the send, so a redelivered job is skipped. Plus provider-side idempotency keys where supported.
“A provider (Twilio) goes down — what happens?” — its per-channel queue backs up, jobs retry with backoff, a per-provider circuit breaker opens to stop hammering it, and for critical channels you fail over to a secondary provider. Email and push are unaffected because they’re separate queues.
“How do you handle quiet hours / preferences?” — the dispatcher consults preferences before fan-out: skip opted-out channels (mark suppressed), and for quiet hours reschedule the send via a delay queue rather than dropping it. Suppression list (from SES bounces/complaints) hard-blocks bad addresses.
“What’s the cost of the async design?” — during a burst, delivery latency rises because jobs queue. That’s the right trade for notifications — late beats lost. Urgent notifications get a separate high-priority queue so they don’t sit behind a marketing blast.