gRPC’s HTTP/2 Basis

gRPC mandates HTTP/2. The features gRPC depends on — multiplexing, server push, header compression — only exist in HTTP/2. Understanding why matters for debugging and capacity planning.

For HTTP fundamentals see ../http/01_http.md (if expanded). Here: what gRPC specifically uses.

HTTP/2 features gRPC relies on

The big ones: multiplexing and streams.

HTTP/2 vs HTTP/1.1

HTTP/1.1:
  Conn A: Request → Response  (next request waits)
  Conn B: Request → Response
  Conn C: ...
  Browsers open ~6 connections per host to parallelize

HTTP/2:
  Conn A: Stream 1: req/resp ─┐
          Stream 3: req/resp ─┤  All concurrent on one connection
          Stream 5: req/resp ─┘

HTTP/1.1’s “one request at a time per connection” was the bottleneck. HTTP/2 multiplexes any number of streams on one TCP connection.

Streams — the unit of work

Each RPC is one HTTP/2 stream:

Client → :method = POST
         :path = /user.UserService/GetUser
         :authority = user-service:50051
         content-type = application/grpc
         te = trailers
         + Protobuf-encoded request body

Server → :status = 200
         content-type = application/grpc
         + Protobuf-encoded response body
         (then trailers)
         grpc-status = 0
         grpc-message = ""

Pseudo-headers (the :method, :path, etc.) are HTTP/2 metadata. Regular headers carry gRPC metadata.

gRPC over HTTP/2 wire format

HEADERS frame  → :method, :path, :authority, content-type, gRPC headers
DATA frame(s)  → request body (one or more, encoded Protobuf)
HEADERS frame  → trailers (grpc-status, grpc-message)

For streaming RPCs, the DATA frames may flow over time:

Client → HEADERS + DATA + DATA + DATA + ... (client streaming continues)
Server → HEADERS + DATA + DATA + DATA + ... (server streaming continues)
       → trailers when done

Trailers are headers sent at the end of the response. HTTP/1.1 doesn’t support trailers reliably; HTTP/2 does. gRPC uses them to communicate status (grpc-status) after the body — important because errors can occur mid-stream.

Why not HTTP/3 / QUIC?

gRPC over HTTP/3 exists but adoption is slow. HTTP/3 uses QUIC (UDP-based) instead of TCP. Pros: avoids TCP head-of-line blocking (a lost packet doesn’t block all streams). Cons: less universally supported, firewall issues with UDP.

For internal service-to-service: HTTP/2 is fine, the network is reliable. For mobile/external: HTTP/3 wins on lossy networks.

TLS — required in practice

HTTP/2 over plain TCP exists (h2c) but is rarely used:

• Browsers won’t speak h2c; require TLS for HTTP/2.
• Most proxies (envoy, nginx) speak h2c only on private interfaces.

In practice, gRPC connections use TLS. With ALPN (Application-Layer Protocol Negotiation), client and server agree on “h2” during the TLS handshake.

For service-to-service inside a VPC: TLS optional (you control the network). For anything across networks: mTLS preferred.

Connection management

One TCP connection per (client, server) pair, shared across many RPCs via multiplexing. Implications:

Load balancing pitfall

HTTP/1.1 + REST: every request is a new connection (or pooled), load balancer can distribute each request to a different backend.

HTTP/2 + gRPC: one TCP connection per backend, many streams on it. If your LB does L4 (TCP), it routes the connection to one backend — all subsequent RPCs go to the same one.

gRPC client → L4 LB → backend A (forever)
gRPC client → L4 LB → backend B (forever)

Hot backends, cold backends, no rebalancing on scale-up.

Solutions:

1. Client-side load balancing: client knows the list of backends, picks one per RPC. gRPC has built-in round_robin policy.
2. L7 LB (Envoy, Linkerd, gRPC-aware nginx): terminates HTTP/2, distributes streams across backends.
3. Headless service in Kubernetes: kubectl get pods returns all pod IPs; gRPC client load-balances.

See 08_load_balancing_grpc_web.md.

Flow control — HTTP/2’s backpressure

Each stream has a window (default 64 KB). The receiver advertises how much it’s willing to accept; the sender stops when the window is full.

Server sends 32 KB → window down to 32 KB
Client processes 16 KB → sends WINDOW_UPDATE(16 KB)
Server can send 16 KB more → window back to 32 KB

Slow consumer = backpressure on the producer automatically. No application code needed.

For high-throughput streams, the default 64 KB window is too small. Tune:

options = [
    ("grpc.http2.lookahead_bytes", 1024 * 1024),
    ("grpc.http2.min_recv_ping_interval_without_data_ms", 5000),
]
channel = grpc.insecure_channel("target", options=options)

Keepalive

HTTP/2 PINGs keep connections alive across firewalls and detect dead peers:

options = [
    ("grpc.keepalive_time_ms", 10000),         # send keepalive every 10s
    ("grpc.keepalive_timeout_ms", 5000),       # if no response in 5s, assume dead
    ("grpc.keepalive_permit_without_calls", 1),
]

Without keepalive, connections idle behind a firewall get silently dropped after some minutes. The next RPC fails with “broken pipe.”

Server-side mirror:

options = [
    ("grpc.keepalive_time_ms", 10000),
    ("grpc.http2.max_pings_without_data", 0),
]
server = grpc.server(executor, options=options)

Debugging HTTP/2

Curl supports HTTP/2:

curl --http2 -k https://localhost:50051/user.UserService/GetUser \
  -H "content-type: application/grpc" -d "..."

But Protobuf payloads aren’t text-friendly. Use:

grpcurl is the standard:

grpcurl -plaintext localhost:50051 user.UserService/GetUser \
  -d '{"id": "42"}'

Requires server reflection (a meta-service that exposes the proto schema) — see 06_interceptors_metadata_auth.md.

Common pitfalls

• L4 load balancer in front of gRPC — all traffic stuck on one backend. Use L7 LB or client-side LB.
• No keepalive — connections silently die behind firewalls; first RPC after idle fails.
• HTTP/2 buffer sizes too small for high-throughput streams — bandwidth limited.
• Connection per RPC — defeats the multiplexing benefit. Use a single channel for the lifetime of the app.
• TLS termination at the wrong proxy — re-terminating gRPC at a proxy that doesn’t speak HTTP/2 breaks everything.

Common interview confusions

• “HTTP/2 uses one TCP connection.” — one per peer typically. You’d have one to each remote service.
• “HTTP/2 server push is widely used.” — almost not at all. The feature was removed from Chrome. gRPC’s “server streaming” is at the application layer, not HTTP/2 server push.
• “L4 load balancing works for gRPC.” — distributes connections, not RPCs. With long-lived connections, work concentrates on a few backends.

Interview angle

• “Why does gRPC require HTTP/2?” — multiplexing (many streams on one TCP conn, no HOL blocking at the request level), header compression (HPACK), trailers (gRPC sends status as trailers after the body — needed for mid-stream errors), flow control per stream.
• “What’s the difference between an HTTP/2 stream and an HTTP/1.1 connection?” — HTTP/2 streams are virtual: many on one TCP connection, framed independently. HTTP/1.1 needs one connection per concurrent request.
• “Why is L4 load balancing a problem for gRPC?” — L4 distributes connections, not streams. gRPC clients use one long-lived connection per backend, so all RPCs land on the same backend. Use L7 (Envoy) or client-side load balancing.
• “How does gRPC handle backpressure?” — via HTTP/2 flow control: per-stream window advertised by receiver, sender stops when full. Slow consumers automatically slow producers. No app code needed; tune window sizes for high-throughput streams.
• “What’s a grpc-status trailer?” — RPC outcome status code, sent as an HTTP/2 trailer (header at end of response). Trailers are needed because errors can happen mid-stream — you can’t put status in regular headers (already sent).
• “How does gRPC keepalive work?” — periodic HTTP/2 PING frames; if no response in keepalive_timeout, the connection is considered dead. Prevents idle connections from being silently dropped by firewalls.