backend / protocols / grpc / 07_error_handling.md

gRPC Error Handling

6 interview angles 6 min read source

gRPC Error Handling

gRPC has its own status codes (not HTTP status codes). 16 well-known codes, returned as trailers after the response. Plus optional rich error details via grpc-status-details-bin.

The 16 status codes

Code Number Meaning
OK 0 success
CANCELLED 1 client cancelled / deadline before completion
UNKNOWN 2 catch-all server error
INVALID_ARGUMENT 3 client sent invalid args (analog of HTTP 400)
DEADLINE_EXCEEDED 4 deadline expired before response (HTTP 504)
NOT_FOUND 5 resource not found (HTTP 404)
ALREADY_EXISTS 6 client tried to create a duplicate (HTTP 409)
PERMISSION_DENIED 7 authenticated but not allowed (HTTP 403)
RESOURCE_EXHAUSTED 8 quota / rate limit hit (HTTP 429)
FAILED_PRECONDITION 9 system not in a state to perform op (HTTP 412)
ABORTED 10 optimistic concurrency conflict (HTTP 409)
OUT_OF_RANGE 11 request specified an invalid range (HTTP 400)
UNIMPLEMENTED 12 method not implemented (HTTP 501)
INTERNAL 13 server bug (HTTP 500)
UNAVAILABLE 14 service temporarily unavailable (HTTP 503)
DATA_LOSS 15 unrecoverable data corruption
UNAUTHENTICATED 16 missing/invalid credentials (HTTP 401)

Memorize the common ones: OK, INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, PERMISSION_DENIED, UNAUTHENTICATED, RESOURCE_EXHAUSTED, UNAVAILABLE, INTERNAL.

Returning errors from a server

Sync — set on context

def GetUser(self, request, context):
    user = db.get_user(request.id)
    if not user:
        context.set_code(grpc.StatusCode.NOT_FOUND)
        context.set_details(f"User {request.id} not found")
        return user_pb2.User()      # empty response (will be discarded)
    return user_pb2.User(id=user.id, name=user.name)

The empty response is ignored when status code is non-OK; status is what the client sees.

Cleaner — abort()

def GetUser(self, request, context):
    user = db.get_user(request.id)
    if not user:
        context.abort(grpc.StatusCode.NOT_FOUND, f"User {request.id} not found")
    return user_pb2.User(id=user.id, name=user.name)

abort() raises an exception that the gRPC framework catches and translates. No need to return anything; control flow stops.

Async equivalent: await context.abort(...).

Handling errors on the client

try:
    response = stub.GetUser(GetUserRequest(id="42"))
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.NOT_FOUND:
        print(f"Not found: {e.details()}")
    elif e.code() == grpc.StatusCode.UNAVAILABLE:
        print("Service unavailable, retrying...")
        retry()
    else:
        raise

grpc.RpcError is the exception type. .code() returns the StatusCode; .details() returns the message.

Rich error details — grpcio-status

For structured error info (not just a string), use google.rpc.Status with details:

from grpc_status import rpc_status
from google.rpc import status_pb2, code_pb2, error_details_pb2

def CreateUser(self, request, context):
    violations = []
    if not request.email or "@" not in request.email:
        violations.append(error_details_pb2.BadRequest.FieldViolation(
            field="email", description="must be a valid email",
        ))
    if not request.name:
        violations.append(error_details_pb2.BadRequest.FieldViolation(
            field="name", description="required",
        ))
    if violations:
        from google.protobuf.any_pb2 import Any
        detail = error_details_pb2.BadRequest(field_violations=violations)
        detail_any = Any()
        detail_any.Pack(detail)
        rich_status = status_pb2.Status(
            code=code_pb2.INVALID_ARGUMENT,
            message="Validation failed",
            details=[detail_any],
        )
        context.abort_with_status(rpc_status.to_status(rich_status))
    ...

Client side:

from grpc_status import rpc_status
from google.rpc import error_details_pb2

try:
    stub.CreateUser(...)
except grpc.RpcError as rpc_error:
    status = rpc_status.from_call(rpc_error)
    for detail in status.details:
        if detail.Is(error_details_pb2.BadRequest.DESCRIPTOR):
            br = error_details_pb2.BadRequest()
            detail.Unpack(br)
            for v in br.field_violations:
                print(f"{v.field}: {v.description}")

The standard error detail types (google.rpc.error_details_pb2):

Type Purpose
BadRequest per-field validation errors
ResourceInfo resource that caused error
RetryInfo how long to wait before retry
DebugInfo stack traces (for internal debugging)
QuotaFailure rate limit / quota details
PreconditionFailure unmet precondition
ErrorInfo structured error code with metadata
Help URL pointing to help
LocalizedMessage translated error

Use these standard types when applicable; build custom messages for app-specific cases.

When to use which status code

The fine distinctions matter for clients building retry logic:

Situation Use Why
Bad input from client INVALID_ARGUMENT client should fix and retry — but with different args
Resource doesn’t exist NOT_FOUND client should not retry; check the id
User not authorized PERMISSION_DENIED client should not retry; need different auth
User not authenticated UNAUTHENTICATED refresh token, retry
Server is down / restarting UNAVAILABLE retry with backoff likely OK
Server bug INTERNAL client can retry but server is broken
Quota exceeded RESOURCE_EXHAUSTED retry with backoff after Retry-After
Deadline expired DEADLINE_EXCEEDED client gave up; retry with longer deadline maybe
Optimistic conflict ABORTED retry after re-reading state
Operation already happened ALREADY_EXISTS idempotent — treat as success or don’t retry

INVALID_ARGUMENT vs FAILED_PRECONDITION: argument validity is per-call; precondition is about system state. “Email format wrong” = INVALID_ARGUMENT. “Bucket must be empty before delete” = FAILED_PRECONDITION.

ABORTED vs UNAVAILABLE: ABORTED suggests “retry will work” (transient state); UNAVAILABLE suggests “service will be back.”

Retry strategy

def call_with_retry(stub, request, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return stub.SomeRpc(request)
        except grpc.RpcError as e:
            if e.code() in (grpc.StatusCode.UNAVAILABLE, grpc.StatusCode.RESOURCE_EXHAUSTED):
                time.sleep(2 ** attempt)        # backoff
                continue
            raise
    raise RuntimeError("retries exhausted")

Retry-safe codes: UNAVAILABLE, RESOURCE_EXHAUSTED, sometimes DEADLINE_EXCEEDED and ABORTED.

Never retry-safe: INVALID_ARGUMENT, NOT_FOUND, PERMISSION_DENIED, UNAUTHENTICATED. The cause is structural; retry will fail the same way.

Built-in retry via service config

service_config = {
    "methodConfig": [
        {
            "name": [{"service": "user.UserService"}],
            "retryPolicy": {
                "maxAttempts": 3,
                "initialBackoff": "0.1s",
                "maxBackoff": "1s",
                "backoffMultiplier": 2,
                "retryableStatusCodes": ["UNAVAILABLE"],
            },
        }
    ]
}
options = [("grpc.service_config", json.dumps(service_config))]
channel = grpc.insecure_channel("server:50051", options=options)

gRPC retries automatically based on this config. Set per-method. The library handles backoff math.

Cancellation and deadlines

# Client — set deadline
try:
    response = stub.GetUser(request, timeout=5.0)
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        ...

After 5 seconds, the RPC is cancelled and DEADLINE_EXCEEDED is raised. The server sees the cancellation via context.is_active().

Deadlines propagate: if A calls B (deadline 5s) and B calls C, C should see how much time remains, not start fresh. gRPC’s metadata carries this automatically when you use the standard interceptors / clients.

Common pitfalls

  • context.set_code(NOT_FOUND) + return user_pb2.User(...) with data — the client sees NOT_FOUND but you sent a valid response. Confusing. Use context.abort or return empty.
  • Returning OK for business errorsgetUser(id="non-existent") → User(id="", name="") (default values). Client sees success with empty data. Confusing; should be NOT_FOUND.
  • Same error code for all failures — clients can’t differentiate “bad input” from “server down” without details. Use specific codes.
  • Retry on INVALID_ARGUMENT — won’t help. Wastes time and resources.
  • No retry on UNAVAILABLE — transient failures become hard failures.
  • ABORTED with no instruction to retry — clients don’t know it’s retryable unless documented.

Common interview confusions

  • “gRPC uses HTTP status codes.” — its own 16 codes. Sent as HTTP/2 trailer grpc-status.
  • “Errors are the response body.” — they’re trailers (headers after the body). Empty body, non-OK status.
  • UNAVAILABLE means the server is permanently down.” — transient by convention. Permanent unavailability would be UNIMPLEMENTED or just a connection failure.

Interview angle

  • “How does gRPC communicate errors?” — gRPC has its own 16 status codes (OK, NOT_FOUND, INVALID_ARGUMENT, UNAVAILABLE, etc.), sent as an HTTP/2 trailer grpc-status after the response. Not HTTP status codes.
  • “Difference between INVALID_ARGUMENT and FAILED_PRECONDITION?” — INVALID_ARGUMENT = bad input that’s wrong regardless of system state (malformed email). FAILED_PRECONDITION = input is fine but the system can’t act on it (bucket must be empty before delete).
  • “Which status codes are retry-safe?”UNAVAILABLE, RESOURCE_EXHAUSTED typically; DEADLINE_EXCEEDED and ABORTED sometimes. Never INVALID_ARGUMENT, NOT_FOUND, PERMISSION_DENIED (the cause won’t change).
  • “How do you return structured error details?” — use grpcio-status to wrap status code + message + a list of detail Protobuf messages (BadRequest, RetryInfo, custom). Clients unpack via rpc_status.from_call.
  • “How does built-in gRPC retry work?” — configure via grpc.service_config channel option: retryPolicy with maxAttempts, initialBackoff, backoffMultiplier, retryableStatusCodes. The library handles backoff and reattempts.
  • context.set_code() vs context.abort()?”set_code sets the status but doesn’t terminate; you must return. abort raises an exception that terminates the handler immediately. abort is cleaner for most cases.