gRPC Load Balancing and gRPC-Web

Two operational topics: how to load balance gRPC traffic (HTTP/2 changes the rules), and how to use gRPC from browsers (which don’t speak it natively).

The load balancing problem

L4 (TCP) load balancers route by connection. With HTTP/1.1 + REST, each request is a new (pooled) connection — naturally distributed.

With gRPC over HTTP/2, the client opens one long-lived TCP connection per backend and multiplexes many RPCs on it. L4 LB picks a backend at connection time; all subsequent RPCs go there:

Client 1 → L4 LB → Backend A (forever)
Client 2 → L4 LB → Backend B (forever)
Client 3 → L4 LB → Backend A (forever)

Adding a new backend C: no existing client uses it. Scale-out doesn’t help under load.

Three solutions

1. Client-side load balancing

Client knows the list of backends, picks one per RPC.

# Round-robin across multiple backends
options = [
    ("grpc.lb_policy_name", "round_robin"),
    ("grpc.service_config", json.dumps({
        "loadBalancingConfig": [{"round_robin": {}}],
    })),
]

# Use dns:/// scheme — gRPC resolves DNS, gets all A records as backends
channel = grpc.insecure_channel(
    "dns:///user-service.acme.svc.cluster.local:50051",
    options=options,
)

The dns:/// scheme tells gRPC to resolve the hostname and treat all returned IPs as backends. Round-robin between them.

In Kubernetes, a headless service (clusterIP: None) makes DNS return all pod IPs:

apiVersion: v1
kind: Service
metadata:
  name: user-service-headless
spec:
  clusterIP: None         # headless
  selector:
    app: user-service
  ports:
    - port: 50051

dns:///user-service-headless resolves to all pod IPs; client-side LB distributes RPCs across them.

Trade-offs:

• True per-RPC balancing.
• No proxy in the path.
• Client knows backend topology — couples client and infra.
• DNS refresh latency for new pods (gRPC default dns_min_time_between_resolutions_ms).

2. L7 (gRPC-aware) load balancer

A proxy that speaks HTTP/2 and distributes streams. Common options:

• Envoy — the standard service mesh data plane. Handles gRPC streams.
• Linkerd — service mesh; transparent L7 LB for gRPC.
• NGINX with grpc_pass — supports gRPC since 1.13.
• HAProxy — gRPC support in recent versions.
• AWS Application Load Balancer (ALB) — supports gRPC.

upstream grpc_backend {
    server pod-1.cluster.local:50051;
    server pod-2.cluster.local:50051;
    server pod-3.cluster.local:50051;
}

server {
    listen 50051 http2;
    grpc_pass grpc://grpc_backend;
}

Trade-offs:

• Client doesn’t know backends; LB is transparent.
• Mature LB features: health checks, retries, circuit breaking.
• Adds a hop and a component to operate.

In service meshes (Istio, Linkerd), Envoy runs as a sidecar — every pod has a transparent gRPC LB.

3. xDS (advanced)

gRPC supports the xDS API directly — the same control-plane protocol Envoy uses. Client gets backend list and policies from a control plane (e.g. Istio’s pilot, Google Traffic Director).

channel = grpc.xds_channel("xds:///user-service")

Combines client-side LB with centralized configuration. Used at scale (Google, large k8s deployments).

Health checking

A backend may be reachable but broken. Health checks let LBs (or clients) skip it.

The gRPC health checking protocol:

service Health {
  rpc Check (HealthCheckRequest) returns (HealthCheckResponse);
  rpc Watch (HealthCheckRequest) returns (stream HealthCheckResponse);
}

message HealthCheckRequest {
  string service = 1;       // "" for overall health
}

message HealthCheckResponse {
  enum ServingStatus { UNKNOWN = 0; SERVING = 1; NOT_SERVING = 2; SERVICE_UNKNOWN = 3; }
  ServingStatus status = 1;
}

Implement in Python:

from grpc_health.v1 import health, health_pb2_grpc

server = grpc.server(...)
health_servicer = health.HealthServicer()
health_pb2_grpc.add_HealthServicer_to_server(health_servicer, server)
health_servicer.set("user.UserService", health_pb2.HealthCheckResponse.SERVING)

LBs (Envoy, NGINX, ALB) can be configured to call the health endpoint. Kubernetes’ grpc-health-probe binary lets readiness/liveness probes do gRPC health checks.

Retries and deadlines at the LB layer

A good LB (Envoy, Linkerd) implements:

• Retries on retryable codes (UNAVAILABLE, RESOURCE_EXHAUSTED).
• Circuit breaking — stop sending to a backend that’s failing.
• Hedging — send to multiple backends, take the first response.
• Outlier detection — eject slow/unhealthy backends.

Doing this at the LB means clients don’t need to implement it; consistent across all callers.

gRPC-Web — gRPC for browsers

Browsers can’t do gRPC directly:

• No native HTTP/2 trailers from fetch() API.
• Browser fetch doesn’t support full HTTP/2 streaming.

gRPC-Web is a wire-compatible adaptation that works over HTTP/1.1 or HTTP/2, parseable from JS.

Browser → gRPC-Web request (HTTP/1.1, base64 or binary)
          ↓
Envoy / grpcwebproxy / Caddy with grpc-web filter
          ↓ translates to native gRPC
Backend (native gRPC server)

Most gRPC servers don’t speak gRPC-Web directly; a proxy translates. Envoy has a built-in envoy.filters.http.grpc_web filter.

Limitations of gRPC-Web

The big one: no client streaming, no bidi. If your service uses those, browsers need a different protocol (REST, WebSocket).

gRPC-Web in Python

You typically don’t write a gRPC-Web server in Python directly. Instead:

1. Write a normal gRPC server.
2. Put Envoy in front with the gRPC-Web filter.
3. Browser uses @improbable-eng/grpc-web or grpc-web JS library.

For pure Python without a separate proxy, sonora is a gRPC-Web-compatible Python framework — useful for prototypes.

pip install sonora

# Sonora exposes ASGI app
from sonora.asgi import grpcASGI
asgi_app = grpcASGI(your_grpc_servicer)

For production: use Envoy/Caddy/nginx proxy + native gRPC server.

Connecting from JS

// @improbable-eng/grpc-web
import { grpc } from "@improbable-eng/grpc-web";
import { UserService } from "./generated/user_pb_service";
import { GetUserRequest } from "./generated/user_pb";

const request = new GetUserRequest();
request.setId("42");

grpc.invoke(UserService.GetUser, {
    request,
    host: "https://api.example.com",
    onMessage: (response) => console.log(response.toObject()),
    onEnd: (code, msg) => console.log("done", code, msg),
});

The JS client generation mirrors Python’s: .proto → JS stubs. Same DX, different runtime.

Alternatives to gRPC-Web

For browser clients, sometimes simpler to:

• REST + gRPC backend: a thin REST gateway translates JSON ↔ Protobuf. grpc-gateway (Go) and envoy gRPC-JSON transcoder automate this from the .proto.
• GraphQL gateway: GraphQL frontend, gRPC backends. Apollo Server + protobufjs.
• WebSocket / SSE: for streaming use cases gRPC-Web can’t handle.

For most browser-facing public APIs: REST is simpler than gRPC-Web. gRPC inside the cluster; REST at the edge.

Common pitfalls

• L4 LB in front of gRPC without realizing it — work concentrates on a few backends. Switch to L7 LB or client-side LB.
• No DNS refresh — client caches DNS forever; new pods aren’t picked up. Set grpc.dns_min_time_between_resolutions_ms lower.
• Not implementing health checks — LBs send traffic to broken backends.
• Trying client/bidi streaming in gRPC-Web — silently fails or doesn’t work. Use server-streaming or fall back to REST/WebSocket.
• gRPC-Web proxy not configured for CORS — browser blocks the call.

Common interview confusions

• “gRPC works in browsers.” — only via gRPC-Web with a translating proxy. Not natively.
• “L4 LB works for gRPC if I add more clients.” — many clients to one LB still concentrate at the connection level. Need L7 or client-side LB.
• “gRPC-Web is a separate protocol.” — wire-compatible adaptation: same .proto, JS client uses a different transport (HTTP/1.1 or HTTP/2 without trailers).

Interview angle

• “Why is L4 load balancing a problem for gRPC?” — HTTP/2 uses one long-lived TCP connection per backend with many multiplexed streams. L4 LB picks the backend at connection time; all subsequent RPCs go there. Scale-out doesn’t redistribute load.
• “What are the options for gRPC load balancing?” — client-side LB (DNS or xDS gives the client a list of backends, picks per RPC), L7 LB (Envoy/NGINX/Linkerd terminates HTTP/2 and distributes streams), or xDS for control-plane-driven LB.
• “How would you load balance gRPC in Kubernetes?” — headless service (returns all pod IPs via DNS) + client-side round-robin LB; or a service mesh (Istio, Linkerd) with Envoy sidecars doing transparent L7 LB.
• “What’s gRPC-Web?” — a wire-compatible adaptation of gRPC for browsers. Works over HTTP/1.1 / HTTP/2 without trailers. Requires a translating proxy (Envoy) in front of native gRPC servers. Limitation: no client or bidi streaming.
• “Why can’t browsers do gRPC directly?” — fetch API doesn’t expose HTTP/2 trailers or full bidirectional streaming. gRPC-Web works around this with a different wire format.
• “What’s the gRPC health checking protocol?” — standardized Health.Check and Health.Watch RPCs (in grpc-health-probe / grpc_health.v1). LBs and k8s probes call them to know whether a backend is ready.