backend / networking / 16_https_handshake_keepalive.md

HTTPS Handshake, mTLS, Keepalive, and the Reverse-Proxy Edge

7 interview angles 8 min read source

HTTPS Handshake, mTLS, Keepalive, and the Reverse-Proxy Edge

The backend-applied layer of networking that the “what is TCP” curriculum often skips. Worth knowing for senior interviews — these are the things that actually fail in production.

HTTPS in two sentences

HTTPS is HTTP over TLS. TLS layers encryption + authentication + integrity onto a raw TCP socket: you know who you’re talking to (server certificate), no one in the middle can read it (encryption), no one can modify it without detection (MAC).

TLS handshake (TLS 1.3)

Client                                Server
  |                                       |
  |--- ClientHello (random, ciphers, ----->|
  |    key share via supported groups)    |
  |                                       |
  |<-- ServerHello (random, cipher,  -----|
  |    key share, cert, finished)         |
  |                                       |
  |    [TLS keys derived from shared      |
  |     secret on both sides]             |
  |                                       |
  |--- Finished, application data ------>|

TLS 1.3 is 1-RTT. One round trip between ClientHello and the first encrypted application data. With 0-RTT (early data resumption) and a previous session, the client can send application data along with the ClientHello — sub-millisecond effective handshake on warm connections.

TLS 1.2 was 2-RTT — half the latency in TLS 1.3.

What you should know about the cert

The server presents an X.509 certificate signed by a CA. Client verifies:

  • Signature chains to a trusted root.
  • subject / SAN matches the requested hostname.
  • notBefore / notAfter are valid (cert not expired).
  • Revocation via OCSP or CRL (often skipped in practice).

A failure on any check → connection aborted. “Untrusted certificate” errors are usually wrong SAN or expired cert.

SNI — Server Name Indication

Multiple HTTPS sites on one IP need to know which cert to present before decrypting the request. SNI is a plaintext extension on the ClientHello: “I’m asking for api.example.com.” The server uses it to pick the right cert.

Caveats:

  • SNI is plaintext — observers see what hostname you’re connecting to. ESNI / ECH (Encrypted Client Hello) encrypts SNI; not yet universally deployed.
  • A misconfigured server may serve the wrong cert; the client gets a hostname mismatch error.

mTLS — Mutual TLS

Standard TLS: client verifies server. mTLS: server also verifies client via a client certificate. Used for:

  • Service-to-service auth. Service A’s cert proves it’s allowed to call service B. The basis of service-mesh identity.
  • High-security APIs where API keys aren’t enough.
  • B2B integrations where the partner provides a cert.
# nginx
ssl_client_certificate /etc/nginx/ca.crt;
ssl_verify_client on;

In a service mesh (Istio, Linkerd), mTLS is the default — sidecars exchange certs automatically; the app sees clear text inside.

TLS termination patterns

Edge termination

Client → LB (TLS terminates) → backend (HTTP)

LB (ALB, nginx, CloudFront) does the TLS work; backend sees plain HTTP. Common, simple, but traffic inside the data center is unencrypted (might be OK if VPC is trusted; otherwise mTLS or re-encryption).

Pass-through

Client → LB (no TLS work) → backend (TLS terminates)

LB forwards encrypted bytes. Backend has the cert. Slower (LB can’t do L7 routing), more complex.

Re-encryption

Client → LB (TLS A) → LB re-encrypts → backend (TLS B)

LB terminates client TLS, opens a new TLS connection to the backend. Encrypted end-to-end with full L7 routing. Higher CPU cost.

For internal services, mTLS via service mesh does the re-encryption pattern automatically.

Keepalive — the connection-reuse story

Opening a TCP+TLS connection is expensive: 3-way TCP handshake + 1-2 RTT TLS handshake = 100-300ms before the first request. HTTP/1.1 added persistent connections (keepalive) — the same TCP connection serves multiple HTTP requests.

HTTP/1.1 200 OK
Connection: keep-alive
Keep-Alive: timeout=60, max=1000

Client and server hold the socket open; subsequent requests skip the handshake. Big latency win for chatty interactions.

Keepalive across a reverse proxy

This is where it gets subtle. Two separate TCP connections:

  • Client ↔ LB (e.g., nginx).
  • LB ↔ Backend pod.

Both have their own keepalive. The LB’s behavior matters for backend performance.

Backend-side keepalive

If nginx opens a new TCP connection to the backend per request, you pay the connect cost every time. Reuse the upstream connection:

upstream backend {
    server backend.svc.cluster.local:8000;
    keepalive 32;          # pool of 32 idle connections per worker
    keepalive_timeout 60s;
    keepalive_requests 1000;
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";    # critical
    }
}

proxy_set_header Connection "" strips the Connection: close that HTTP/1.0 would default to; needed for upstream keepalive. Without it, nginx closes the upstream connection after each request and you’ve defeated the keepalive.

Idle connection timeouts

client ←→ LB ←→ backend
       60s    300s

If client keepalive timeout (60s) is shorter than LB-to-backend timeout (300s), and the LB doesn’t notice client disconnection, the LB may try to use a closed socket → reset; either side sees Connection reset by peer.

The cure: align timeouts so the backend’s idle close > LB idle close > client idle close. Some LBs proactively probe.

Backend pod and load balancer interactions

In Kubernetes with ALB target-type=ip + readiness probes:

1. Pod terminates: gets SIGTERM
2. Application stops accepting new requests (readiness fails)
3. ALB removes pod from rotation (takes a few seconds — health-check based)
4. In-flight requests drain
5. Pod exits

The window between SIGTERM and “ALB removed from rotation” is the source of 502s. preStop hook with 5-10s sleep gives the LB time to notice. See 17_kubernetes/02_pods_services_deployments.md.

HTTP/2 and HTTP/3

HTTP/2

  • Single TCP connection, many multiplexed streams.
  • Header compression (HPACK).
  • Server push (rarely used).

Removes “head-of-line blocking on the HTTP layer” (HTTP/1.1 pipelining is barely deployable). One slow response on a stream doesn’t block others (within the same connection).

But TCP head-of-line blocking still exists: a lost TCP packet pauses everything until retransmitted. HTTP/3 fixes this.

HTTP/3

  • Built on QUIC (UDP-based).
  • Each stream is independent — lost packet only stalls its own stream.
  • 0-RTT for repeat connections.
  • Mandatory TLS 1.3 inside QUIC.

Browsers and major CDNs use HTTP/3. Many backends still default to HTTP/2 or HTTP/1.1; supported but less common than you’d expect.

NAT and the connection problem

Behind NAT (your residential router, AWS NAT Gateway, K8s ClusterIP):

  • Outbound connections work fine; the NAT box rewrites source IP.
  • Long-idle connections may have their NAT mapping evicted; subsequent packets dropped.

NAT timeouts:

  • AWS NAT Gateway: 350 seconds idle. Below this, connections stay alive; above, mappings drop. Watch for idle DB connections, long-poll HTTP, WebSockets.
  • Residential routers: often 60-300s.

For long-lived connections, send periodic keepalive packets to refresh the NAT mapping. TCP keepalive is too infrequent by default (2 hours); application-level pings (every 30s) are better.

Reverse-proxy gotchas

Connection: close from the backend kills keepalive

If your backend sends Connection: close, the proxy closes the upstream connection. No reuse. Costs connect-time on the next request. Don’t set Connection: close unless you mean it.

Buffered vs streaming responses

Default nginx buffers the upstream response before sending to the client. Useful for slow clients (frees backend faster) but breaks Server-Sent Events / streaming responses. Disable for those:

location /stream {
    proxy_pass http://backend;
    proxy_buffering off;
    proxy_read_timeout 86400;
}

Slow clients holding connections

Without buffering, a slow client holds the backend connection until they consume the response. A few thousand slow clients can exhaust backend connection pools. nginx’s default buffer protects you here.

X-Forwarded-For

The client’s real IP isn’t visible to the backend (it sees the LB’s IP). The LB adds an X-Forwarded-For header:

X-Forwarded-For: 203.0.113.5, 10.0.1.5

First value = original client; subsequent = proxies. Trust only the count you control; an attacker can forge earlier values. Most frameworks have a setting for “trust N hops” (e.g., uvicorn forwarded_allow_ips).

Common production failures

  • Cert expired. Set a calendar alert; better — automate renewal via cert-manager / Let’s Encrypt.
  • Hostname mismatch. Cert has SAN api.example.com but client connects to api.staging.example.com.
  • Mixed-content errors. HTTPS page loading HTTP resources. Browser blocks. Fix: serve all assets via HTTPS.
  • Backend reset after idle. NAT timeout, connection pool eviction. Add app-level pings, or shorter pool TTL.
  • 502 on rolling deploy. SIGTERM → app exits before LB removes from rotation. Add preStop sleep.
  • Slow client backpressure. Without buffering, slow client holds backend connection. Tune.

Interview angle

  • “Walk through the TLS handshake.” — TLS 1.3 is 1-RTT: ClientHello (random + ciphers + key share), ServerHello (key share + cert + finished), then encrypted application data. Cert is verified for signature, hostname (SAN), validity dates, revocation. TLS 1.2 was 2-RTT.
  • “What’s SNI and why does it matter?” — Server Name Indication. Plaintext extension on ClientHello indicating which hostname you’re connecting to. Lets a server with one IP host many certs. Plaintext = observable; ECH encrypts it but isn’t universally deployed.
  • “What’s mTLS?” — Mutual TLS: both sides present certificates. Server verifies the client’s cert as well. Used for service-to-service auth (the basis of service mesh identity) and high-security APIs.
  • “How does keepalive across a reverse proxy work?” — two separate TCP connections (client↔LB and LB↔backend). Each has its own keepalive. nginx needs explicit upstream { keepalive 32; } + proxy_http_version 1.1; proxy_set_header Connection ""; to reuse upstream connections.
  • “Why does your service get 502s on rolling deploys?” — SIGTERM → app exits → LB still routing → request hits dead pod. Fix: app should fail readiness on SIGTERM but keep serving for a brief window (preStop sleep) until LB removes from rotation, then drain in-flight.
  • “HTTP/2 vs HTTP/3?” — HTTP/2 multiplexes streams over one TCP connection (removes HTTP-layer head-of-line blocking, but TCP-layer still exists). HTTP/3 runs over QUIC (UDP), each stream independent, lost packet only stalls its own stream. 0-RTT resumption built in.
  • “What’s the NAT timeout problem?” — long-idle connections may have their NAT mapping evicted; subsequent traffic dropped. AWS NAT Gateway = 350s. Fix: app-level keepalive pings (~30s interval) to refresh mappings.
Previous
Ingress vs Egress Traffic