Inter-service Communication

Two big buckets: synchronous (caller waits) and asynchronous (fire-and-forget through a broker). The first interview question is almost always “when do you pick which?”

Synchronous: HTTP/REST and gRPC

Caller blocks on a response. Coupling is temporal — both services must be up at the same time.

REST/JSON over HTTP

import httpx

async def get_user(user_id: int) -> dict:
    async with httpx.AsyncClient(timeout=2.0) as c:
        r = await c.get(f"http://users/v1/users/{user_id}")
        r.raise_for_status()
        return r.json()

Pros: ubiquitous, debuggable (curl), browser-friendly, easy versioning. Cons: verbose payloads, no schema enforcement unless you add OpenAPI, slower than binary protocols.

gRPC

Protobuf-defined contracts, HTTP/2, bidi streaming.

# Stub generated from orders.proto
import grpc, orders_pb2, orders_pb2_grpc

async def place_order(order):
    async with grpc.aio.insecure_channel("orders:50051") as ch:
        stub = orders_pb2_grpc.OrdersStub(ch)
        return await stub.Place(orders_pb2.PlaceRequest(...))

Pros: binary + smaller, strong schema, streaming, deadlines built in, polyglot. Cons: browser support requires gRPC-Web, less human-debuggable, harder local testing.

Rule of thumb

REST for external APIs, partner integrations, browser-facing. gRPC for internal service-to-service where latency and contracts matter.

Asynchronous: message brokers and events

Caller publishes a message; consumer processes later. Spatial coupling stays, temporal coupling drops — the consumer can be down for a minute and you don’t care.

Command vs Event

Mixing the two is a classic distributed-system smell. If consumers are explicitly listed in the sender’s code, it’s a command pretending to be an event.

Pattern: pub/sub with SNS + SQS

publisher → SNS topic ─┬─> SQS queue (orders consumer)
                       ├─> SQS queue (analytics consumer)
                       └─> SQS queue (audit consumer)

Each consumer has its own queue subscription, retries independently, and can dead-letter without affecting the others. This is the AWS canonical pattern.

Pattern: Kafka topic

Same idea but consumers pull at their own pace, can replay, and ordering is preserved per-partition. See backend/10_message_queues/kafka/.

The synchronous-async decision matrix

Anti-patterns

• Synchronous chain ≥ 4 services deep. Latency multiplies, every link is a failure point. Refactor to async fan-out or merge services.
• HTTP “events” via polling. If you find yourself polling /orders?since=..., you want a queue/topic.
• Distributed monolith via shared library. Services that can’t deploy independently are a monolith with extra steps.
• Async for in-line UX. If the user is waiting on a button, async + “we’ll email you” works; async + UI silently waiting doesn’t.

Timeouts, retries, backoff

Mandatory for every sync call. Defaults are not defensible.

async with httpx.AsyncClient(
    timeout=httpx.Timeout(connect=1.0, read=2.0, write=2.0, pool=5.0)
) as c:
    ...

Retry rules:

• Only retry idempotent operations (GET, PUT with idempotency key, POSTs with explicit Idempotency-Key header).
• Exponential backoff with jitter — min(cap, base * 2**attempt) + random(0, base).
• Retry 5xx and connect errors; don’t retry 4xx (the bug is on your end).
• Cap total attempts (3 is common).

Circuit breaker

When the dependency is dying, stop hammering it. Three states:

CLOSED  (normal) ──failures over threshold──> OPEN  (fast-fail, no calls)
              ▲                              │
              │                          after cooldown
              │                              │
              └──── HALF_OPEN ◄──────────────┘
                  (probe; success → CLOSED, fail → OPEN)

Libraries: purgatory, pybreaker, or the breaker inside tenacity/stamina. Often the sidecar (Envoy/Istio) does this for you transparently.

Schema evolution and versioning

Async messages live longer than the code that emitted them. Rules:

• Add fields, don’t rename or remove for a long deprecation window.
• Use Protobuf or Avro with a schema registry for strict cross-team contracts.
• For JSON: tolerant readers (ignore unknown fields) + producers that never omit existing fields.
• Version in the URL path for sync (/v1/...), in the message envelope for async (event_version: 2).

Interview angle

• “Sync vs async — when each?” — sync when the caller needs the answer in the same request; async when the work is a side-effect, or the dependency can be slow/unreliable, or you need fan-out.
• “You have Order → Payment → Inventory → Email all over HTTP. What’s wrong?” — synchronous chain; tail-latency multiplies; one slow service kills the user request. Refactor: Order publishes OrderPlaced; payment, inventory, email subscribe.
• “How do you retry safely?” — only on idempotent ops; exponential backoff with jitter; don’t retry 4xx; cap attempts.
• “REST or gRPC for internal services?” — gRPC where you control both ends and latency matters; REST for external/partner APIs and ad-hoc debugging.
• “What does a circuit breaker do that retries don’t?” — retries protect a single call from a transient blip; breaker protects the dependency (and your threadpool) from collapse during a sustained outage.