Kafka: Log Compaction, Rebalancing, and Advanced Topics

The two operational topics that come up most often in senior Kafka interviews — and that the basic “consumer groups + offsets” coverage doesn’t address.

Log compaction

Standard Kafka retention is time-based (retention.ms — keep for 7 days) or size-based (retention.bytes). Older messages get deleted.

Log compaction is a different cleanup policy: keep the latest message per key, garbage-collect older ones. Topic acts like a key-value snapshot rather than a log.

# Create a compacted topic
kafka-topics --create \
  --topic user-state-snapshot \
  --partitions 12 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.1

key=user-42  value=v1  (compacted away)
key=user-42  value=v2  (compacted away)
key=user-42  value=v3  ← kept (latest)
key=user-99  value=v1  ← kept

After compaction, the topic holds the most-recent state per key forever (unless you retention.ms it).

When to use

• Change Data Capture / state snapshots. “Current state per user_id” — newcomers can replay the topic from beginning to rebuild state.
• Configuration / lookups. Topic as a slow-update KV store.
• Stream-table joins in Kafka Streams. The “table” side is a compacted topic.

Tombstones

A null value for a key is a tombstone — log compaction deletes the key entirely after delete.retention.ms. Used to remove keys from the state.

Trade-offs

• Cleaner can’t compact log segments that are still being written. Active segments stay non-compacted; only sealed segments get compacted.
• Compaction needs CPU; tune log.cleaner.threads.
• Replaying a compacted topic from start gives the latest state quickly (vs replaying a non-compacted log forever).

Cleanup policies can combine

cleanup.policy=compact,delete
retention.ms=2592000000   # 30 days

Compact within the retention window; delete keys older than 30 days. Both at once. Used when you want the latest-per-key for recent data and bounded total storage.

Rebalancing — the consumer group dance

Consumer groups partition the topic among members. When membership changes (new consumer, dead consumer, scaled up/down, deploy), Kafka rebalances — re-assigns partitions across the current members.

The basic algorithm (eager): all consumers stop, all assignments revoked, new plan computed, partitions reassigned, consumers resume. Brief pause for the whole group during the dance. For a busy consumer group, this can be 5-30 seconds of stalled processing.

Sticky assignor

Default since Kafka 2.4. Tries to keep existing assignments stable across rebalances — a consumer that had partitions 3, 7, 11 before the rebalance keeps them after (when possible). Minimizes re-warming of consumer state.

Cooperative rebalancing

Newer (Kafka 2.4+, CooperativeStickyAssignor). Two-phase rebalance:

1. Members report current assignments; revoke ONLY the partitions that need to move.
2. Reassign just those partitions.

The rest of the consumers keep processing throughout. Massively reduces stop-the-world pauses.

Enable:

consumer = KafkaConsumer(
    "orders",
    group_id="processor",
    bootstrap_servers="...",
    partition_assignment_strategy=[CooperativeStickyAssignor],
)

Both producers and consumers in the group must agree on the strategy. Mixed assignors fall back to eager.

The new consumer group protocol (KIP-848)

Everything above describes the classic, client-coordinated protocol. As of Kafka 4.0 (March 2025) there is a second one, and it’s the answer that shows you’re current.

KIP-848 moves the assignment logic from the client to the broker. The group coordinator computes the assignment and pushes each member its share; there’s no join/sync barrier where everyone waits for the slowest member, and no requirement that all clients agree on an assignor.

consumer = KafkaConsumer(
    "orders",
    group_id="processor",
    bootstrap_servers="...",
    group_protocol="consumer",     # opt in; "classic" is still the client default
)

As of 2026-08: GA in Apache Kafka and Confluent Cloud; early access in librdkafka-based clients (2.10+), which matters for confluent-kafka-python.

Interview angle: “how do you reduce rebalance impact?” — the layered answer is (1) cooperative sticky assignor on the classic protocol, (2) tune session.timeout.ms / max.poll.interval.ms so slow processing doesn’t look like a dead consumer, (3) on 4.x, move to group.protocol=consumer and let the broker do incremental assignment. Naming KIP-848 signals you’ve touched Kafka recently.

Static membership

Consumer specifies a stable group.instance.id. On disconnect (e.g., a brief network blip, a deploy), the broker doesn’t immediately rebalance — it waits session.timeout.ms for that instance to come back, keeping its partitions reserved.

consumer = KafkaConsumer(
    "orders",
    group_id="processor",
    group_instance_id="processor-pod-3",
    session_timeout_ms=30_000,
)

Useful for stateful consumers (where re-warming state costs more than tolerating brief unavailability). Combine with session.timeout.ms = 30000 for a 30s tolerance window.

Causes of rebalance

• Consumer joins or leaves — graceful or crashed.
• max.poll.interval.ms exceeded — consumer is alive but not polling; broker assumes it’s stuck and kicks it.
• session.timeout.ms exceeded — heartbeats not received.

The most common production cause: long message processing exceeds max.poll.interval.ms (default 5 min). Consumer is alive but appears stuck. Either speed up processing, raise max.poll.interval.ms, or use the pause/resume API to take more time without losing membership.

Exactly-once semantics (EOS)

Producers can be idempotent: a retry of the same message doesn’t duplicate. Combined with transactional writes (one transaction writing to multiple topic-partitions), you get exactly-once within Kafka.

producer = KafkaProducer(
    bootstrap_servers="...",
    enable_idempotence=True,
    acks="all",
    max_in_flight_requests_per_connection=5,
    transactional_id="my-app-1",
)
producer.init_transactions()

producer.begin_transaction()
producer.send("output", value=result1)
producer.send("audit", value=audit_record)
producer.send_offsets_to_transaction(consumer.position(), consumer.groupMetadata())
producer.commit_transaction()

send_offsets_to_transaction is the key: the consumer’s offset commit is part of the same transaction as the producer writes. Either all-or-none — exactly-once consume-process-produce.

Caveats:

• Adds latency (~10-20ms per transaction).
• Requires isolation.level=read_committed on downstream consumers.
• Doesn’t extend to external systems (DB writes are NOT in the Kafka transaction; that’s the dual-write problem — use outbox / Debezium).

Producer acks and durability

producer = KafkaProducer(acks="all", retries=10, max_in_flight_requests_per_connection=5)

• acks=0 — fire and forget. Lowest latency, highest data loss risk.
• acks=1 — leader broker writes to log + returns. Loss if leader dies before replication.
• acks=all — wait for all in-sync replicas (ISR) to confirm. Highest durability.

With min.insync.replicas=2 topic config and acks=all, you need ≥2 replicas alive to accept writes. Trade durability for availability.

ISR (In-Sync Replicas)

A partition has N replicas; the ones caught up within replica.lag.time.max.ms are “in-sync.” A replica that falls too far behind is removed from ISR temporarily.

Leader election picks from ISR. If unclean.leader.election.enable=true, the broker may elect a non-ISR replica — fast failover but data loss (the non-ISR replica is missing recent writes). Production default: unclean.leader.election.enable=false.

Common gotchas

• auto.offset.reset=latest on first-run. New consumer starts from “latest” → skips everything written before subscribing. Set to earliest for new consumer groups that should backfill.
• enable.auto.commit=true. Commits happen periodically without regard to whether the message was processed. Switch to manual commits + commit after work succeeds for at-least-once.
• Long processing + small max.poll.interval.ms. Triggers rebalance during message handling. Workers churn; nothing makes progress.
• Single partition for high-throughput topic. All writes serialize on one partition; one consumer max. Plan partitions up front; you can add but not reduce them.
• Repartitioning a key-sharded topic. The key→partition mapping changes; messages with the same key may end up on different partitions across the change. Be careful with state.
• Re-reading from start of compacted topic. OK, but you get tombstones too (for delete.retention.ms window) — your consumer must handle the null value as “delete this key from my state”.

Interview angle

• “What’s log compaction in Kafka?” — cleanup policy that keeps only the latest message per key, garbage-collects older versions. Topic acts like a KV snapshot; replaying from start gives the current state per key. Used for CDC / state snapshots / lookup topics.
• “How does Kafka rebalance work?” — when consumer group membership changes, partitions are reassigned. Default (sticky) tries to keep assignments stable. Cooperative rebalancing (2.4+) revokes only the partitions that need to move, instead of stopping the whole group.
• “What’s a tombstone?” — null value for a key in a compacted topic. Signals deletion; the compactor removes that key entirely after delete.retention.ms. Used to evict state.
• “How do you get exactly-once with Kafka?” — exactly-once delivery doesn’t exist over the network; exactly-once processing is achieved by idempotent producers + transactional writes + send_offsets_to_transaction. Doesn’t extend to external systems — use outbox / CDC for “exactly-once across DB + Kafka”.
• “What’s an ISR?” — In-Sync Replicas: replicas caught up within replica.lag.time.max.ms. Only ISR members are eligible for leader election. min.insync.replicas=2 + acks=all requires 2 ISR replicas to accept writes — durability over availability.
• “What causes a consumer rebalance?” — member joins/leaves the group, session.timeout.ms exceeded (heartbeats missed), max.poll.interval.ms exceeded (processing took too long without polling). The last one is the most common production cause.
• “What’s static membership?” — consumer has a stable group.instance.id. On disconnect, the broker reserves its partitions for session.timeout.ms instead of immediately rebalancing. Useful for stateful consumers across brief disruptions like rolling deploys.