Event-Driven Architecture and Sagas

Two related architectural styles for distributed systems. Event-driven architecture decouples services by communicating through asynchronous messages. Sagas coordinate multi-step distributed transactions when ACID transactions across services aren’t possible.

For EDA as a standalone overview see 19_event_driven_architecture.md. For CQRS/Event Sourcing see 13_cqrs_event_sourcing.md. For microservices context see 08_monolith_vs_microservices.md.

Event-Driven Architecture (EDA)

Services don’t call each other directly. They publish events; other services subscribe.

┌──────────┐     OrderPlaced      ┌──────────┐
│ Order    │ ──────────────────►  │ Email    │
│ Service  │                       │ Service  │
└──────────┘                       └──────────┘
     │                                 ↓
     │  OrderPlaced
     ↓
┌──────────┐                       ┌──────────┐
│ Inventory│ ◄─────────────────►  │ Analytics│
│ Service  │                       │ Service  │
└──────────┘                       └──────────┘

Producer publishes OrderPlaced; multiple consumers react independently. Adding a new consumer (e.g., a new “loyalty points” service) doesn’t change the producer.

The vocabulary

The key distinction: events describe what happened (past tense, immutable); commands describe an intent (imperative).

Why EDA

Patterns within EDA

Event Notification: producer says “thing happened”; consumer fetches details from the producer’s API if needed.

OrderPlaced(order_id=42)  ←  just the ID
Email service: GET /orders/42 to fetch details

Small events. Consumer load on the producer’s API. The producer’s data is authoritative.

Event-Carried State Transfer: event contains all the data consumers need.

OrderPlaced(order_id=42, customer_email="...", items=[...], total=...)
Email service: has everything, doesn't call producer

Larger events. No coupling on producer’s API. Consumers can be more autonomous.

Event Sourcing: events ARE the state. See 13_cqrs_event_sourcing.md.

Choreography: services coordinate via events, no central conductor. Orchestration: a central conductor (orchestrator) tells services what to do.

These last two are about coordinating multi-step workflows — leading to sagas.

Sagas — distributed transactions without ACID

In a monolith, multi-step operations use a database transaction:

with transaction.atomic():
    debit_account(from_acc, amount)
    credit_account(to_acc, amount)
    log_transfer(from_acc, to_acc, amount)
# atomic: all or none

Across microservices each with its own database, you can’t span them in one ACID transaction. The CAP / 2PC problem makes distributed transactions painful and rare.

Saga: a sequence of local transactions, with compensating transactions to undo earlier steps if a later one fails.

1. Reserve inventory               → if fail, abort
2. Charge payment                  → if fail, release inventory
3. Create shipment                 → if fail, refund payment, release inventory
4. Send confirmation               → if fail, no rollback needed (idempotent retry)

Each step is a local transaction in one service. Failure at step N triggers compensating transactions for steps 1 to N-1.

Choreography saga

No central coordinator. Services react to each other’s events.

Order Service: PlaceOrder → emits OrderPlaced
Inventory Service: hears OrderPlaced → reserves stock → emits StockReserved (or StockFailed)
Payment Service: hears StockReserved → charges → emits PaymentCharged (or PaymentFailed)
Shipping Service: hears PaymentCharged → creates shipment → emits ShipmentCreated
Order Service: hears ShipmentCreated → marks order complete

If PaymentFailed: Inventory Service hears it → releases stock; Order Service marks failed.

Pros:

• No single point of coordination.
• Services are decoupled.

Cons:

• Hard to follow the workflow — logic is scattered.
• Adding a step means modifying multiple services.
• Cyclic dependencies emerge (“who handles the failure?”).

Good for simple workflows. Becomes spaghetti for complex ones.

Orchestration saga

A central orchestrator coordinates the steps.

Order Service receives PlaceOrder
  → calls Orchestrator
  → Orchestrator: ReserveStock command to Inventory
  → Inventory: replies StockReserved
  → Orchestrator: ChargePayment command to Payment
  → Payment: replies PaymentCharged
  → ...

If PaymentFailed:
  → Orchestrator: ReleaseStock compensating command to Inventory
  → Orchestrator: marks saga as failed

Pros:

• Workflow logic in one place — readable.
• Easier to add/remove steps.
• Centralized error handling.

Cons:

• Orchestrator is a single point of (logical) failure.
• Orchestrator must be available + scaled appropriately.
• Some coupling between services and orchestrator.

Most complex workflows use orchestration. Tools: Temporal, AWS Step Functions, Camunda, Netflix Conductor.

Compensating actions

Compensations are NOT just “undo” in the SQL sense. They’re business-level reversals:

Compensations can fail too. The orchestrator must handle that (retry, alert, manual intervention).

Some actions can’t be compensated (sending a notification, transferring physical goods). Order those LAST so failure doesn’t require undoing them.

Idempotency — non-negotiable

Distributed events are delivered with at-least-once semantics. Consumers WILL see duplicate events. Without idempotency, every event you process N times is N times the side effects.

# Idempotency via processed-event log
def handle_order_placed(event):
    if processed.exists(event_id=event.id):
        return       # already handled
    with transaction.atomic():
        do_the_work(event)
        processed.create(event_id=event.id, processed_at=now())

Or via natural idempotency keys:

def handle_payment_charged(event):
    # Insert with conflict-on-duplicate; the unique constraint makes this safe
    PaymentLog.objects.update_or_create(
        order_id=event.order_id,
        defaults={"amount": event.amount, "status": "charged"},
    )

Every consumer must be safe to invoke multiple times with the same input. Without this, EDA falls apart at the first network blip.

Message brokers

Kafka has become the default for event sourcing and high-throughput pipelines. RabbitMQ is still common for task queues and slower-moving event flows. Managed services (SNS/SQS, Pub/Sub) for cloud-native.

Outbox pattern — solving the “two writes” problem

A service that writes to its DB AND publishes an event has a consistency problem: what if the DB succeeds and the event publish fails (or vice versa)?

# Naive — fragile
def place_order(order):
    db.insert(order)              # writes DB
    broker.publish(OrderPlaced(order.id))    # if this fails, event is lost

Outbox: write the event to an “outbox” table in the same transaction as the business write. A separate process reads the outbox and publishes to the broker.

def place_order(order):
    with db.transaction():
        db.insert(order)
        db.insert_outbox(OrderPlaced(order.id))
    # commit guarantees both happen or neither

# Separate worker:
while True:
    events = db.fetch_outbox_batch()
    for event in events:
        broker.publish(event)
        db.mark_outbox_processed(event.id)

The outbox + a worker is the canonical pattern for “ensure event publishing is consistent with DB writes.” Variants:

• Transactional outbox + polling worker.
• CDC (Change Data Capture) reading from the DB log (Debezium).
• Two-phase commit (rare, complex).

Common pitfalls

• No idempotency: duplicate event = duplicate side effect.
• Synchronous “events”: producer waits for all consumers — defeats decoupling. Events should be fire-and-forget.
• Tight coupling via event shapes: producer changes the event payload; all consumers break. Use schema evolution (Avro, Protobuf with optional fields, JSON Schema with backward-compat).
• Lost events: writing DB then publishing (or vice versa) without outbox/CDC.
• Out-of-order events: ordering only guaranteed within a partition (Kafka) or queue (Rabbit). Cross-partition order isn’t.
• Choreography for complex flows: spaghetti. Use orchestration past 3-4 steps.
• Compensations that don’t actually compensate: charging $100 and refunding $99 because of fees. Compensations must reach an acceptable end state.
• Saga with no timeout: a step stalls; saga waits forever. Each step has a timeout + failure path.

Common interview confusions

• “Events and commands are the same.” — events are facts (past tense); commands are intents (imperative). Events: “OrderPlaced.” Commands: “PlaceOrder.”
• “Saga = distributed transaction.” — saga is a workflow with compensations, NOT atomic. Each local step commits independently. Compensations roll back as a sequence, not atomically.
• “EDA means microservices.” — EDA works inside a monolith too (modules communicating via events). Microservices benefit from it but don’t require it.

Interview angle

• “What is event-driven architecture?” — services communicate by publishing and subscribing to events (asynchronous, past-tense facts) via a broker, rather than direct synchronous calls. Decouples producers from consumers; enables independent scaling, resilience, and extensibility.
• “Event vs command?” — event: “something happened” (past, fact, multiple consumers). Command: “do this” (imperative, single intended recipient). Events go through pub/sub brokers; commands are usually direct calls (or via a queue with one consumer).
• “What’s a saga?” — a sequence of local transactions across distributed services, with compensating transactions to undo earlier steps if a later one fails. Replaces ACID transactions for cross-service operations.
• “Choreography vs orchestration saga?” — choreography: services react to each other’s events, no central coordinator (decoupled but distributed logic). Orchestration: a central orchestrator coordinates each step (centralized logic, single coordinator). Complex workflows usually use orchestration.
• “What’s a compensating transaction?” — business-level reversal of a forward action (refund a charge, release reserved stock). Not the same as SQL rollback; they’re new operations that bring the system to an acceptable state.
• “Why is idempotency critical in event-driven systems?” — events are delivered with at-least-once semantics. Consumers see duplicates. Without idempotency, duplicate events cause duplicate side effects (double charges, etc.). Track processed-event IDs or use natural idempotency keys.
• “What’s the outbox pattern?” — write events to a DB “outbox” table in the same transaction as the business write. A separate worker reads the outbox and publishes to the broker. Solves the “DB write succeeds but event publish fails” consistency problem.
• “When does EDA not pay off?” — simple synchronous workflows where the latency and reasoning cost of async event flows isn’t justified. Monolithic apps with one DB don’t need EDA for transactions; they have ACID.