Nginx Upstreams and Load Balancing

The upstream block defines a pool of backend servers. proxy_pass http://<upstream-name> routes to that pool. Nginx handles the algorithm, health, and connection reuse.

Basic upstream

upstream app {
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
    server 10.0.0.3:8000;
}

server {
    location / {
        proxy_pass http://app;
    }
}

Default algorithm: round robin.

Algorithms

upstream app {
    least_conn;
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
}

upstream cache_pool {
    hash $request_uri consistent;        # cache locality — same URI to same backend
    server 10.0.0.10:8000;
    server 10.0.0.11:8000;
    server 10.0.0.12:8000;
}

For most Python apps with similar request costs: round robin or least_conn. least_conn wins when request durations vary a lot.

Per-server options

upstream app {
    server 10.0.0.1:8000 weight=3;                      # 3x as many requests
    server 10.0.0.2:8000 weight=1;
    server 10.0.0.3:8000 backup;                        # only used when others are down
    server 10.0.0.4:8000 down;                          # explicitly disabled
    server 10.0.0.5:8000 max_fails=3 fail_timeout=30s;  # health
    server 10.0.0.6:8000 max_conns=100;                  # cap concurrent connections
}

How health checks work (open source nginx)

Open-source nginx does passive health checks only:

• A request fails (timeout / 5xx / connect refused) → counted as a fail.
• After max_fails consecutive fails, the server is marked down for fail_timeout.
• After fail_timeout, nginx tries again; success = back in rotation; fail = down for another fail_timeout.

There’s no out-of-band “ping the health endpoint” probe in open-source nginx. For active health checks you need:

• NGINX Plus (commercial): health_check uri=/healthz interval=5s; directive.
• Third-party module: nginx_upstream_check_module.
• External tool: cron job that calls each backend and re-renders nginx config + reload.
• Service mesh / cloud LB: skip nginx health checks, use Envoy / ALB / HAProxy in front.

What counts as a fail

proxy_next_upstream controls which upstream errors trigger fail counting and request retry:

proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
proxy_next_upstream_tries 3;
proxy_next_upstream_timeout 10s;

Defaults:

• error timeout (always retry on these).
• 5xx and invalid_header are NOT default — opt in.

Don’t add non_idempotent unless you want POST/PUT to be retried (which can cause duplicate operations — be careful).

Connection keepalive to upstreams

By default, nginx opens a fresh TCP connection to the upstream for each request. For high-RPS workloads this is wasteful — every request pays a TCP handshake.

upstream app {
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
    keepalive 32;                # max idle connections per worker per upstream
    keepalive_requests 1000;     # max requests on one connection before re-opening
    keepalive_timeout 60s;       # idle timeout
}

server {
    location / {
        proxy_pass http://app;
        proxy_http_version 1.1;          # required for keepalive
        proxy_set_header Connection "";  # MUST clear; default is "close"
    }
}

Without proxy_http_version 1.1 and Connection "", keepalive won’t engage. This is a common “I added keepalive but it’s not helping” bug.

For Python upstreams (Gunicorn, uWSGI), keepalive can dramatically reduce per-request overhead at high RPS.

Connection limits per backend

upstream app {
    zone app 64k;
    server 10.0.0.1:8000 max_conns=50;
}

zone enables shared memory across workers (so the connection count is global, not per-worker). Without zone, max_conns is per-worker — easy to exceed without realizing.

Multiple upstreams for routing

upstream api { server 10.0.0.10:8000; }
upstream web { server 10.0.0.20:8080; }
upstream admin { server 10.0.0.30:8000; }

server {
    location /api/ { proxy_pass http://api; }
    location /admin/ { proxy_pass http://admin; }
    location / { proxy_pass http://web; }
}

Each pool can have its own algorithm, weights, keepalive settings.

DNS resolution gotcha

If server lists a hostname (not IP), nginx resolves it at startup or reload. Hostname changes (e.g. cloud autoscaling, container restarts) won’t be picked up:

upstream app {
    server app.internal:8000;        # resolved once at start
}

For dynamic DNS:

resolver 10.0.0.2 valid=30s;         # nginx's DNS resolver, with 30s cache TTL

server {
    location / {
        set $upstream "app.internal";
        proxy_pass http://$upstream:8000;
    }
}

Using a $variable in proxy_pass triggers nginx to re-resolve based on the resolver’s TTL. Catch: this disables the URI-replacement behavior; combine with rewrite if needed (see 05_proxy_pass_trailing_slash.md).

NGINX Plus has resolve parameter on server that handles this without the variable trick.

stream block — TCP/UDP load balancing

For non-HTTP services (Postgres, Redis, raw TCP):

stream {
    upstream pg_pool {
        server 10.0.0.50:5432;
        server 10.0.0.51:5432;
    }

    server {
        listen 6432;
        proxy_pass pg_pool;
        proxy_timeout 30s;
    }
}

L4 load balancing — nginx distributes connections, not requests. A long-lived database connection sits on one backend forever. For even spread under that pattern, use connection pooling on the app side (pgbouncer) and short-lived TCP connections.

See ../../28_networking/10_load_balancers.md for L4 vs L7 trade-offs.

Common interview confusions

• “Open-source nginx does active health checks.” — only NGINX Plus / third-party modules / external tools. OSS does passive only (failed requests count toward max_fails).
• “Adding keepalive to upstream is enough.” — also need proxy_http_version 1.1 and proxy_set_header Connection "". Otherwise keepalive doesn’t engage.
• “Hostnames in server re-resolve as DNS changes.” — only at startup/reload. Use resolver + $variable in proxy_pass, or NGINX Plus resolve parameter.
• “ip_hash is real session affinity.” — basic. Doesn’t survive backend changes, doesn’t handle clients behind NAT (all routed to the same backend). For real stickiness use hash $cookie_session consistent.

Interview angle

• “Walk me through nginx’s load balancing options.” — round robin (default), least_conn (best for variable request durations), ip_hash / hash (stickiness), random / random two. Per-server: weight, max_fails, fail_timeout, backup, max_conns.
• “How does nginx do health checks?” — open source: passive only — count failures, mark down after max_fails, retry after fail_timeout. Active health checks need NGINX Plus or external tooling.
• “Why isn’t keepalive to my upstream working even though I configured it?” — missing proxy_http_version 1.1 and proxy_set_header Connection "". Keepalive requires HTTP/1.1 and clearing the default Connection: close header.
• “How does proxy_next_upstream work?” — controls which conditions (error, timeout, http_500, etc.) cause nginx to retry on the next upstream and count toward max_fails. Be careful adding non-idempotent retries — POST/PUT can duplicate.
• “My backend’s IP changed but nginx still sends to the old one — why?” — hostnames are resolved at startup/reload. Use resolver and a $variable in proxy_pass to re-resolve at request time.
• “How would you do consistent hashing in nginx?” — hash $key consistent; in the upstream block. Useful for cache locality (same key always to same backend; minimal reshuffle when backend pool changes).