Kafka delivery semantics — at-most-once, at-least-once, exactly-once

“What delivery guarantee does Kafka give?” is a trap question: Kafka gives you whichever you configure, and each side (producer, consumer) contributes independently. This note is the decision space; broker-side durability (acks, ISR) is covered in 02_log_compaction_rebalance.md.

The three semantics

Almost every real system runs at-least-once + idempotent processing. Exactly-once is a specific Kafka feature with a specific scope — not magic.

Producer side

Three settings interact:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "broker:9092",
    "acks": "all",               # wait for all in-sync replicas
    "enable.idempotence": True,  # dedupe broker-side on retry
    # implied by idempotence: retries > 0, max.in.flight <= 5
})

• acks=0 — don’t wait at all → at-most-once (loss on any hiccup).
• acks=1 — leader only → lost if leader dies before followers copy.
• acks=all — all in-sync replicas → durable, pair with min.insync.replicas=2.

The retry duplicate problem: with acks=all and retries, a write can succeed but the ack get lost; the producer retries and the broker appends the message twice. That’s why at-least-once produces duplicates even when your code sends once.

Idempotent producer (enable.idempotence=true) fixes exactly this case: each producer gets a PID, each message a sequence number per partition; the broker drops re-sends it has already appended. Since Kafka 3.0 it’s the default. Scope limits worth saying in an interview:

• Dedupes only broker-side retries within one producer session — not two produce() calls from your code, not app restarts.
• Per partition, per producer. It is not an application-level dedupe.

Transactions — the “exactly-once” machinery

Transactions make a set of writes (to multiple partitions) and an offset commit atomic:

producer = Producer({
    "bootstrap.servers": "broker:9092",
    "transactional.id": "order-enricher-1",   # stable per logical producer
})
producer.init_transactions()

while True:
    msgs = consumer.consume(100, timeout=1.0)
    producer.begin_transaction()
    for m in msgs:
        producer.produce("enriched-orders", transform(m.value()))
    producer.send_offsets_to_transaction(          # offsets commit INSIDE the txn
        consumer.position(consumer.assignment()),
        consumer.consumer_group_metadata())
    producer.commit_transaction()

Key mechanics:

• transactional.id must be stable across restarts. On restart the broker bumps the producer epoch and fences the zombie — an old instance still running can’t commit anymore. This is what makes “process each message once” survive crashes.
• Downstream consumers must set isolation.level=read_committed or they’ll see aborted transactional writes.
• The unit being made atomic is consume → transform → produce: the input offset and the output records commit or abort together.

Scope — the part interviewers probe: Kafka EOS covers Kafka-to-Kafka pipelines. The moment your consumer writes to Postgres, calls Stripe, or sends an email, Kafka transactions do not reach there. End-to-end you’re back to:

• idempotent writes / natural keys / upserts in the sink,
• idempotency keys (../../13_architecture_design/18_idempotency_keys.md, ../celery/04_idempotency.md),
• or the outbox pattern on the producing side (../../13_architecture_design/16_transactional_outbox.md).

“Exactly-once delivery” doesn’t exist over arbitrary side effects; exactly-once processing is engineered with dedupe.

Consumer side — offsets decide your semantics

The consumer’s guarantee is set by when you commit relative to processing:

enable.auto.commit=true commits on a timer (default 5s) from the polling thread — meaning offsets for messages you haven’t finished processing can be committed (loss), or long processing can crash after side effects but before the timer (duplicates). You get vague semantics; that’s why serious consumers commit manually:

consumer = Consumer({
    "bootstrap.servers": "broker:9092",
    "group.id": "billing",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",   # where to start with NO committed offset
})
consumer.subscribe(["orders"])
while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    handle(msg)                        # side effects first
    consumer.commit(msg, asynchronous=False)  # then commit — at-least-once

• commit(asynchronous=True) is higher-throughput but a failed commit is only logged; commit sync at least on shutdown/rebalance.
• Commit granularity is per partition offset, not per message — committing offset N says “everything up to N is done.”
• auto.offset.reset only applies when the group has no committed offset (new group, or offsets expired) — a classic incident cause: latest + expired offsets silently skips data.

Rebalance interplay: when partitions are revoked (02_log_compaction_rebalance.md), commit what you’ve finished in the revoke callback, or the next owner reprocesses your in-flight batch. With cooperative-sticky rebalancing the disruption is smaller but the rule is the same: uncommitted = redelivered.

Choosing, quickly

Common pitfalls

• Trusting enable.idempotence to dedupe application-level re-sends — it doesn’t.
• Auto-commit + slow handlers: offsets committed for unprocessed messages → silent loss.
• Forgetting read_committed downstream of a transactional producer → consumers read aborted data.
• One transactional.id shared by multiple instances → constant fencing errors.
• Measuring “exactly-once” by absence of duplicates in a happy-path test — the duplicates appear only on retries/rebalances/crashes.

Interview angle

• “What delivery semantics does Kafka support and how do you get each?” — Producer (acks, idempotence, transactions) × consumer (commit before/after processing). Name the crash windows.
• “How does the idempotent producer work?” — PID + per-partition sequence numbers; broker drops duplicates from retries; session-scoped only.
• “Is Kafka exactly-once real?” — Yes, for Kafka-to-Kafka via transactions + zombie fencing + read_committed; end-to-end with external systems you add idempotency/outbox. Saying “yes, everywhere” or “no, it’s a lie” are both wrong.
• “What happens to uncommitted offsets during a rebalance?” — Redelivery to the new assignee; commit on revoke; design handlers to be idempotent anyway.