Load balancing and CDN

The first two boxes in almost every system design diagram. Interviewers check whether you can say why each choice, not just draw them.

Load balancing layers

L4 forwards packets. Very fast, protocol-agnostic, and it cannot route on a URL path or terminate TLS meaningfully. Use it for raw throughput, non-HTTP protocols, or when you need to preserve the client IP end to end.

L7 terminates the connection and understands HTTP. That buys path-based routing, header manipulation, TLS termination, request-level retries, canary splitting and compression — at the cost of more CPU and higher latency per request.

Most web systems want L7. Reach for L4 when you’re proxying gRPC streams at volume, a database protocol, or anything where the proxy shouldn’t understand the payload.

Algorithms

Least connections is usually the better default than round robin. Round robin assumes every request costs the same; one slow endpoint on a round-robin pool means the unlucky backend accumulates work while others idle.

Consistent hashing deserves its own note: hashing key mod N remaps almost every key when N changes. Consistent hashing maps both keys and nodes onto a ring so adding or removing a node only moves 1/N of keys. That’s what makes it right for cache sharding — see ../../backend/08_databases/sql/14_sharding_partitioning.md.

Health checks

The part that decides whether your load balancer helps or hurts.

• Passive — mark a backend unhealthy after N consecutive failures on real traffic. Free, but real users see those failures.
• Active — poll an endpoint on a schedule. Costs traffic, detects problems before users do.

Make the health endpoint shallow by default. A check that queries the database means a database blip marks every backend unhealthy simultaneously, and the load balancer removes your entire fleet. That’s a correlated-failure amplifier, and it’s a real outage pattern.

Distinguish the two Kubernetes-style probes, because they behave differently:

Putting a dependency check in liveness is how a slow database becomes a restart loop. See ../../backend/17_kubernetes/04_probes_and_hpa.md.

Sticky sessions, and why to avoid them

Session affinity pins a client to one backend. It works and it creates problems: uneven load, lost sessions on deploy, and it blocks autoscaling from helping the hot instance.

The better answer is stateless backends with session state in Redis or a signed token. Then any request can go anywhere, deploys are trivial, and scaling works.

Say that explicitly — “I’d make the backend stateless rather than use sticky sessions” is a stronger design answer than configuring affinity well.

CDN

A geographically distributed cache in front of your origin.

What it buys:

• Latency — content served from a nearby edge instead of one region.
• Origin offload — cache hits never reach you, which is the main cost saving.
• DDoS absorption — the edge takes the volume.
• TLS termination close to the user, cutting handshake round trips.

Caching correctly

Cache-Control: public, max-age=31536000, immutable    # hashed asset filenames
Cache-Control: public, max-age=60, stale-while-revalidate=300
Cache-Control: private, no-store                       # per-user content

The two patterns worth knowing:

Content-hashed filenames plus immutable. app.a3f9c1.js never changes, so cache it for a year. A deploy produces a new filename, so there’s nothing to invalidate. Cache invalidation stops being a problem by construction — that’s the design insight, and it’s better than any purge strategy.

stale-while-revalidate serves stale content immediately while refreshing in the background. Users never wait on a revalidation, and the origin sees far fewer synchronous requests.

Purging is slow and eventually consistent across a global network. Design so you rarely need it: version URLs rather than invalidating them.

Cache keys

By default the key is the URL. Anything that varies the response must be in the key or you serve the wrong content to someone.

Vary: Accept-Encoding is fine. Vary: Cookie effectively disables caching, since every user has a distinct cookie. If content genuinely varies per user, it isn’t CDN-cacheable — split the page into a cacheable shell and a per-user fragment fetched separately.

Beyond static assets

Modern CDNs cache API responses and run code at the edge (Cloudflare Workers, Lambda@Edge). Edge compute suits auth checks, redirects, A/B assignment and personalisation headers — work that’s cheap, stateless and latency-sensitive.

Don’t put your database there. Edge compute is far from your data, so anything requiring a round trip to origin storage is slower at the edge, not faster.

Interview angle

• “L4 or L7 load balancer?” — L7 for HTTP, because you get path routing, header manipulation, TLS termination and request-level retries. L4 when you need raw throughput, a non-HTTP protocol, or the proxy shouldn’t inspect the payload.
• “Which balancing algorithm?” — least connections as a default over round robin, because round robin assumes uniform request cost. Consistent hashing when you need cache affinity, since it only remaps 1/N of keys when the pool changes.
• “How do you health check?” — active checks on a shallow endpoint. A deep check that hits the database will mark every backend unhealthy at once during a database blip and remove your whole fleet. Keep dependency checks out of liveness probes specifically, or a slow dependency causes restart loops.
• “How do you handle sessions behind a load balancer?” — make the backend stateless: session state in Redis or a signed token. Sticky sessions cause uneven load, break on deploy and undermine autoscaling.
• “How do you invalidate CDN content?” — mostly you don’t. Content-hashed filenames with immutable mean a deploy produces new URLs, so there’s nothing to purge. Purging is slow and eventually consistent, so design to avoid needing it.
• “Can you cache a personalised page on a CDN?” — not as one object. Vary: Cookie effectively disables caching. Split it: a cacheable shell plus a per-user fragment fetched client-side or assembled at the edge.