backend / microservices / 04_data_consistency_patterns.md

Data Consistency Across Services

5 interview angles 5 min read source

Data Consistency Across Services

The hard part of microservices isn’t splitting code — it’s splitting state. Each service owns its DB, so the old solution (“just do a transaction”) doesn’t exist. Several patterns cover the gap.

The problem

@db.transaction
def place_order(order):
    orders.insert(order)        # ours
    inventory.reserve(...)      # someone else's API/DB — not in our tx
    payments.charge(...)        # someone else's API/DB — not in our tx

If payments.charge fails after inventory.reserve succeeded, who unreserves? No 2PC across HTTP. Distributed transactions (XA) exist in theory; almost no one uses them at scale.

Pattern 1 — Saga

Replace the distributed transaction with a sequence of local transactions, each with a compensating action in case a later step fails.

Choreography (event-driven)

Each service listens for events and emits its own.

OrderService:   create_order(PENDING) → emit OrderCreated
PaymentService: on OrderCreated → charge → emit PaymentSucceeded | PaymentFailed
InventoryService: on PaymentSucceeded → reserve → emit InventoryReserved | InventoryUnavailable
OrderService:   on InventoryReserved → mark CONFIRMED
                on PaymentFailed or InventoryUnavailable → mark FAILED + emit OrderFailed
PaymentService: on OrderFailed → refund (compensating)
  • Pros: no central coordinator, services are autonomous.
  • Cons: flow is implicit — hard to see “what happens when order is placed” without grepping every service.

Orchestration

A central orchestrator service explicitly calls each step.

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order):
        charge_id = await execute(payments.charge, order)
        try:
            await execute(inventory.reserve, order)
        except ActivityError:
            await execute(payments.refund, charge_id)   # compensating
            raise
        return Confirmation(charge_id)
  • Pros: flow is visible in one place; easy to add new steps.
  • Cons: orchestrator becomes a god service; some coupling back.

Tools: Temporal, AWS Step Functions, Camunda, Netflix Conductor. See backend/10_message_queues/temporal/.

Saga rules

  • Every step that has side effects needs a compensation.
  • Compensations must be idempotent — they run after retries and may run more than once.
  • Sagas provide atomicity (all-or-nothing semantically) but not isolation — other readers see intermediate states. Handle with semantic locks (e.g., status=PENDING) or read-after-commit.

Pattern 2 — Transactional Outbox

The trickiest atomicity problem: write to your DB and publish an event. If the DB commits and the publish fails, the event is lost; if the publish succeeds and the DB rollback, you publish a phantom event.

# WRONG — dual write
def place_order(order):
    db.commit_order(order)
    kafka.publish("OrderPlaced", order)   # crash here → lost event

Fix: write the event into the same DB transaction as the business write, in an outbox table. A separate process reads the outbox and publishes.

BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox  (aggregate, event_type, payload, created_at)
       VALUES ('order', 'OrderPlaced', :payload, now());
COMMIT;
# outbox_relay.py — runs in its own process
def relay():
    while True:
        rows = db.fetch("SELECT id, payload FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED")
        for r in rows:
            kafka.publish("OrderPlaced", r.payload)     # at-least-once
            db.execute("UPDATE outbox SET published_at = now() WHERE id = :id", id=r.id)

Properties:

  • Local transaction is the source of truth.
  • The relay can crash and retry; events are at-least-once.
  • Consumers must be idempotent (see below).

Alternative: Change Data Capture (CDC) via Debezium reads the DB’s write-ahead log directly and emits events to Kafka — no outbox table, no relay process. Heavier infra, less code.

Pattern 3 — Idempotent consumers + idempotency keys

At-least-once delivery means duplicates. Every consumer must dedupe.

Idempotency key in the producer

POST /charges
Idempotency-Key: 7f3a-...
{ "amount": 1000 }

Server stores (key, response) in a table. Same key retried → return stored response, do nothing.

CREATE TABLE idempotency_keys (
    key         TEXT PRIMARY KEY,
    request_hash TEXT NOT NULL,         -- detect mis-reuse with different payload
    response    JSONB,
    status_code INT,
    created_at  TIMESTAMPTZ DEFAULT now()
);
@app.post("/charges")
async def charge(payload: ChargeIn, idempotency_key: str = Header(...)):
    existing = await db.fetchone("SELECT * FROM idempotency_keys WHERE key = :k", k=idempotency_key)
    if existing:
        if existing["request_hash"] != hash(payload):
            raise HTTPException(409, "Idempotency-Key reused with different payload")
        return JSONResponse(existing["response"], status_code=existing["status_code"])

    result = await do_charge(payload)
    await db.execute("INSERT INTO idempotency_keys ...", ...)
    return result

Dedupe at the consumer

Store (source, message_id) after processing; check first.

CREATE TABLE processed_messages (
    source       TEXT NOT NULL,
    message_id   TEXT NOT NULL,
    processed_at TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (source, message_id)
);

The trick: insert into processed_messages in the same transaction as the side effect, so duplicates fail the unique constraint and roll back.

Pattern 4 — Eventual consistency + read-your-writes

After write to A, the read replica or the projection in service B is eventually consistent. UIs that immediately read-back may show stale data.

Mitigations:

  • Read-your-writes: UI uses the version returned by the write for the next read; service waits until that version is visible.
  • Optimistic UI: show the write as committed even if backend is still catching up.
  • Version vectors / ETags: clients pass the version they expect; backend serves at-least-that-version.

Pattern 5 — Aggregate boundaries (DDD)

Choose service boundaries so most transactions are local. If Order and OrderItem are always changed together, they belong to the same aggregate in the same service. If Order and Inventory change independently, separate services are fine.

Symptom of bad boundaries: every business operation requires distributed transactions or sagas. Fix the boundaries; don’t add more sagas.

Anti-patterns

  • Dual-write without outbox. “Write to DB then publish to Kafka” loses messages on every crash.
  • Cross-service joins via shared DB. Bypasses the service boundary; if you can read another service’s tables, you’re a distributed monolith.
  • Synchronous chain pretending to be a saga. A 5-deep HTTP call chain with no compensations is not a saga; one failure halfway = inconsistent state.
  • Two-phase commit across HTTP. Don’t. The few systems that need it use Spanner-like infra.

Quick decision flow

Need atomicity across services?
├── Can you change the boundary so it's local?    → Do that.
├── Is "eventually consistent" acceptable?         → Saga (choreography) + outbox.
├── Need visible flow / human-in-loop / long-running? → Saga (orchestration via Temporal/Step Functions).
└── Truly need ACID across services?              → Reconsider design; or accept Spanner-class infra.

Interview angle

  • “How do you do a transaction across two services?” — you don’t. Either move the boundary, or use a saga (orchestrated for long-running, choreographed for simple event chains) with compensating actions, plus idempotency in every consumer.
  • “What’s the dual-write problem?” — atomically writing to two systems (DB + queue, DB + cache, DB + Kafka). The classic fix is the transactional outbox: write the event into a same-DB outbox table inside the business transaction; a relay reads and publishes. Alternative: CDC via Debezium.
  • “At-least-once vs exactly-once?” — exactly-once delivery doesn’t exist over a lossy network; exactly-once processing does, achieved by at-least-once delivery + idempotent consumers. Idempotency is the actual primitive.
  • “Choreography vs orchestration — when each?” — choreography for short flows where each service is autonomous; orchestration when you need visibility, complex branching, or long-running with human steps.
  • “Why is 2PC rarely used in microservices?” — coordinator is a SPOF, blocks participants, fails badly under partition, doesn’t compose with HTTP. The industry settled on sagas + idempotency instead.
Previous
Distributed Tracing