system_design / worked designs / 05_chat_presence.md

Worked Design — Chat & Presence

6 interview angles 6 min read source

Worked Design — Chat & Presence

“Design a chat system” (Slack/WhatsApp-style). Tests WebSocket-at-scale, the connection-routing problem, message ordering/delivery, and presence. Stack: FastAPI + WebSockets + Redis + Postgres/Cassandra + SQS.

1. Requirements

Functional: 1:1 and group messages; real-time delivery to online recipients; offline users get messages on reconnect; delivery/read receipts; online/offline presence.

Non-functional: low latency for delivery (sub-second to online users); durable (a delivered message is never lost); ordered within a conversation; scale to millions of concurrent connections.

Clarify: group size limits? message history retention? receipts required or nice-to-have? E2E encryption (usually out of scope for the design round)?

2. Scale

50M DAU, ~10M concurrent connections at peak
each user sends ~40 messages/day → 2B messages/day → ~23,000 writes/sec avg
delivery fan-out: each message goes to 1 (DM) or N (group) recipients

The headline number: 10M concurrent persistent connections. A connection is held open continuously — this is the dominant infrastructure concern.

3. The connection layer — WebSockets

HTTP request/response can’t push; chat needs WebSockets (persistent, bidirectional). See ../../backend/12_protocols/websockets/ for the protocol depth.

Client ──WebSocket──► Load Balancer ──► WebSocket Gateway servers (stateful: hold connections)

Key facts:

  • Each gateway server holds tens-of-thousands of open connections (bounded by file descriptors, memory per connection).
  • 10M connections ÷ ~50k per server ≈ 200 gateway servers.
  • Connections are sticky — once established, that client stays on that server (the LB must support sticky/long-lived connections; an NLB or a WebSocket-aware ALB).
  • Gateway servers are stateful (they hold connections) — but the application logic should stay stateless; the gateway is a thin routing layer.

4. The core problem — routing a message to a recipient’s connection

User A (on gateway server 3) sends a message to user B. B’s connection is on some other server — which one? This is the central design challenge.

Connection registry — a Redis hash mapping user_id → gateway_server_id (and connection id). Updated on connect/disconnect.

A sends msg → gateway 3 → look up B in registry: "B is on gateway 17"
            → publish the message to gateway 17 (via Redis Pub/Sub or a message bus)
            → gateway 17 pushes it down B's WebSocket

So the flow uses a pub/sub backplane between gateway servers: each gateway subscribes to a channel for the users/conversations it holds; delivering to a remote user means publishing to that user’s gateway. Redis Pub/Sub works at moderate scale; at very large scale a dedicated bus (Kafka) or a purpose-built routing tier.

For group chats: the sender’s gateway looks up all members, groups them by gateway server, and publishes once per gateway (not once per user).

5. Message flow — durability + delivery together

A message must be persisted before it’s considered accepted, and delivered to whoever’s online:

1. A's gateway receives the message.
2. Persist to the message store (this is the durability point — now it can't be lost).
3. Ack back to A ("sent").
4. Fan out to recipients:
     - online (in connection registry) → route via backplane → push down their WebSocket → "delivered"
     - offline → it's already in the store; they'll fetch it on reconnect
5. Recipient's client sends a read receipt → another tiny message back through the system.

The ordering matters: persist, then ack, then deliver. If you ack before persisting and the server crashes, the sender thinks it’s sent but it’s gone.

6. Message storage

Access pattern: “get the last N messages in conversation X, in order” and “get messages in conversation X since timestamp T” (for reconnect catch-up).

  • Partition key: conversation_id. Sort key: message timestamp (or a sequence number).
  • This is a textbook Cassandra / DynamoDB access pattern — high write throughput, always queried by conversation_id, time-ordered within. Postgres works at smaller scale (partition the messages table by conversation or time), but the write volume (~23k/sec) and the clean key-based access pattern favor a wide-column / KV store.
  • Ordering within a conversation: don’t trust client timestamps (clock skew) or wall-clock alone. Assign a per-conversation sequence number server-side, or use the storage layer’s clustering order. Within one conversation, messages must be totally ordered; across conversations, order doesn’t matter.

7. Presence

“Is user X online?” — looks simple, is fiddly:

  • On connect, set presence:{user_id} = online in Redis with a TTL (e.g. 30s).
  • The client sends a heartbeat every ~15s; each heartbeat refreshes the TTL.
  • If heartbeats stop (crash, network drop), the key expires → user shows offline. This handles ungraceful disconnects, which a simple connect/disconnect flag would miss.
  • Broadcasting presence changes to everyone is expensive at scale — only push presence updates to users who are actually viewing that person (an open conversation, a visible contact list), not globally.

Presence is best-effort and approximate — a few seconds of staleness is fine; don’t over-engineer it into a strongly-consistent system.

8. Offline delivery & reconnect

When a user reconnects:

  1. Client says “last message id/timestamp I have for each conversation.”
  2. Server queries the message store for everything newer, per conversation.
  3. Push the backlog down the new connection.

Because every message was persisted at step 2 of the send flow, offline delivery is just “read from the store on reconnect” — no separate offline-queue needed (though a per-user “undelivered” pointer can speed up the catch-up query).

9. Bottlenecks & trade-offs

  • Connection count is the cost center — 10M connections is ~200 stateful servers doing little CPU but holding lots of memory/FDs. They must shed and re-balance connections gracefully (a deploy can’t drop 50k connections hard — clients reconnect, but it’s a thundering herd).
  • The backplane — Redis Pub/Sub is simple but every gateway sees every message on subscribed channels; at extreme scale you need smarter routing (consistent-hash users to gateways, or a dedicated routing service). State Redis Pub/Sub as the v1, the dedicated bus as the scale answer.
  • Group chat fan-out — a 10k-member group message fans out to 10k deliveries. Group by gateway server to minimize backplane publishes; very large groups (broadcast channels) may switch to a fan-out-on-read model like the news feed.
  • Reconnect storms — a gateway server dying drops all its connections at once; they reconnect simultaneously. Mitigate with jittered client reconnect backoff and gateway capacity headroom.
  • Ordering vs availability — strict per-conversation ordering needs a sequencing point; under partition you may have to choose. For chat, slight reordering of near-simultaneous messages is usually tolerable — don’t over-pay for it.
  • WebSocket vs alternatives — for a system that’s mostly server→client (live notifications, not chat), SSE is simpler (plain HTTP, auto-reconnect). Chat genuinely needs bidirectional → WebSockets.

Interview angle

  • “A sends a message to B — how does it reach B’s connection?” — B’s WebSocket is on some gateway server. A connection registry (Redis: user_id → gateway_server) tells A’s gateway where B is; the message is published across a pub/sub backplane to B’s gateway, which pushes it down B’s socket. Group chat: group recipients by gateway, publish once per gateway.
  • “How do you guarantee a message is never lost?” — persist to the message store before acking the sender. Order is persist → ack → deliver. Offline recipients fetch from the store on reconnect — the store is the offline queue.
  • “What store for messages and why?” — partitioned by conversation_id, sorted by time — a Cassandra/DynamoDB-shaped access pattern with high write throughput. Postgres (partitioned) works at smaller scale. Ordering within a conversation comes from a server-assigned sequence number, not client timestamps.
  • “How do you handle 10M concurrent connections?” — stateful WebSocket gateway servers, ~50k connections each, ~200 servers; sticky load balancing for long-lived connections; the gateway is a thin routing layer while app logic stays stateless. Graceful connection draining on deploy to avoid reconnect storms.
  • “How does presence work?” — Redis key per user with a short TTL, refreshed by client heartbeats. Stop heartbeating (crash, drop) → key expires → offline. Best-effort and approximate; only broadcast presence changes to users actually viewing that person, not globally.
  • “WebSocket or SSE for this?” — WebSocket: chat is genuinely bidirectional (send and receive). SSE would be the choice if it were server→client only (live feed, notifications) — simpler, plain HTTP, auto-reconnect — but it can’t carry the client’s outbound messages.
Previous
Worked Design — News Feed / Timeline