Load Balancers

A load balancer fronts a pool of backends and distributes incoming requests across them. It also does health checks (remove dead backends), TLS termination, and often more (rate limiting, WAF, sticky sessions).

In a Python web stack you typically have at least one load balancer in front of your app servers — even if you only run one app server today, you’ll need it tomorrow.

L4 vs L7 — the central distinction

If you’re load-balancing HTTP APIs, default to L7 (ALB / nginx). Only drop to L4 when you’re not doing HTTP or you need maximum throughput / TLS passthrough.

Algorithms

Real LBs (nginx, Envoy, ALB) default to round robin or least connections. The fancier algorithms matter less than people think — the actual lever is good health checks plus a sane backend pool.

Health checks

The LB pings each backend periodically and removes failing ones from the pool. Tunables:

Best practice for the health endpoint:

• Cheap — should not query the database. A /healthz that hits Postgres on every check makes the LB part of your DB outage.
• Liveness vs readiness — liveness = “process is alive”; readiness = “ready to serve”. Two endpoints, two behaviors. (Kubernetes formalized this; the same idea applies to LBs.)
• Return JSON, not HTML — easier to parse in dashboards.

TLS termination

The load balancer decrypts incoming HTTPS, sees plain HTTP, makes a routing decision, and either forwards as plain HTTP (most common) or re-encrypts to the backend (TLS re-encryption).

For most Python web apps: terminate at LB, plain HTTP backend, both inside a private VPC. mTLS / passthrough is for compliance-heavy environments.

Sticky sessions (session affinity)

Without affinity, a user’s requests can hit any backend. If sessions are in cookies/JWT/Redis, fine. If sessions are in process memory (flask.session with the in-process dev backend), affinity is mandatory.

Two common strategies:

Avoid sticky sessions when you can. They prevent rolling deploys from gracefully draining traffic and make scale-up uneven.

Connection draining / graceful shutdown

When you remove a backend (deploy, scale-in), the LB should:

1. Stop sending it new connections (deregister).
2. Wait for in-flight requests to finish (drain timeout, e.g. 30s).
3. Then it’s safe to terminate.

Configurable in ALB (“deregistration delay”), nginx, Kubernetes (terminationGracePeriodSeconds). Without this, deploys cause request errors.

Multi-AZ and HA

A single LB endpoint should fan out to backends in multiple AZs. AWS ALB/NLB are inherently multi-AZ — you pick the subnets and they place LB nodes in each AZ. DNS returns multiple IPs (one per AZ).

Health checks per-AZ: if one AZ goes dark, that AZ’s LB nodes are removed from DNS rotation (slow — DNS TTL).

AWS LB choices

ALB is the default for HTTP. NLB is for non-HTTP, ultra-low latency, or when you need a static IP per AZ.

nginx as a load balancer

upstream app {
    least_conn;
    server app1.internal:8000 max_fails=3 fail_timeout=30s;
    server app2.internal:8000;
    server app3.internal:8000 backup;
}

server {
    listen 443 ssl http2;
    location / {
        proxy_pass http://app;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

X-Forwarded-For matters: behind an LB, request.META["REMOTE_ADDR"] is the LB’s IP — use X-Forwarded-For to get the real client. (Django needs SECURE_PROXY_SSL_HEADER and USE_X_FORWARDED_HOST configured.)

Layer 4 with HAProxy / NLB

For TCP services without HTTP awareness (Postgres, Redis, gRPC streams):

[Client] ──TCP:6432──▶ [HAProxy/NLB] ──TCP:5432──▶ [pgbouncer pool]

L4 LBs can’t read the request, so they distribute by connection (not request). Means a long-lived connection sits on one backend forever — not great if you want even load. Solve with shorter connection lifetimes or app-side connection pooling.

Common interview confusions

• “L7 LB is always slower than L4.” — measurably yes, but in microseconds. Networking around it dominates. Pick by features, not theoretical latency.
• “Sticky sessions == HA.” — opposite, often. Sticky breaks rolling deploys and makes scale-out uneven. Externalize session state instead.
• “LB IP is static.” — depends. ALB has a DNS name with rotating IPs (use the DNS name, not IPs). NLB has static IPs per AZ.
• “Health check should test everything.” — no. Should test that the process can serve a basic response. If you check the DB and the DB blips, the LB pulls every backend out and you have a worse outage.

Interview angle

• “L4 vs L7 load balancer — when do you reach for which?” — L7 (ALB/nginx) for HTTP — content-based routing, TLS termination, header-aware. L4 (NLB/HAProxy-TCP) for non-HTTP, very high throughput, or TLS passthrough.
• “How does an LB know a backend is alive?” — periodic health checks (HTTP path or TCP connect). Multiple consecutive failures remove it; multiple consecutive successes re-add it. Tune intervals/thresholds for failure-detection speed vs flapping.
• “What goes in a /healthz endpoint?” — minimal: process is up and can return 200. Avoid downstream dependencies (DB, cache) — coupling the LB’s “is this alive” decision to a flaky downstream amplifies outages.
• “Sticky sessions — when needed, when avoid?” — needed when session state is in process memory; avoid otherwise (breaks rolling deploys, uneven scaling). Move sessions to Redis/JWT/cookies.
• “What does X-Forwarded-For do?” — header set by the LB carrying the original client IP, since the backend sees the LB’s IP as REMOTE_ADDR. Frameworks need explicit config to trust it (otherwise spoofable).
• “How does TLS termination work and what are the trade-offs?” — LB decrypts incoming TLS, talks plain HTTP to the backend (simplest), or re-encrypts (mTLS/compliance), or passes TCP through (LB never sees plaintext, no L7 routing).