What is Kafka?

Definition

Apache Kafka is a distributed streaming platform designed to handle high-throughput, fault-tolerant, real-time data streaming. It was originally developed by LinkedIn and is now an open-source project under the Apache Software Foundation.

Key Concepts

Core Components

1. Producer: Applications that publish (write) messages to topics
2. Consumer: Applications that subscribe to topics and process the stream of messages
3. Topic: A category or feed name to which messages are published
4. Partition: Topics are divided into partitions for parallelism and scalability
5. Broker: A Kafka server that stores topic data
6. Cluster: A group of brokers working together
7. Consumer Group: A group of consumers that work together to consume a topic
8. KRaft Controller: Manages cluster metadata, broker membership, and leader election using an internal Raft quorum. This is the only coordination mechanism as of Kafka 4.0.
9. Zookeeper (removed): Coordinated the cluster in Kafka < 4.0. Fully removed in Kafka 4.0 (March 2025) — do not describe it as current.

Architecture

Producer → Topic (Partitions) → Consumer Group
              ↓
           Broker 1
           Broker 2
           Broker 3

When to Use Kafka

• Real-time data streaming: Processing streams of events in real-time
• Event sourcing: Storing events as they happen
• Microservices communication: Decoupled service-to-service messaging
• Log aggregation: Collecting logs from multiple services
• Metrics collection: Gathering metrics from distributed systems
• Activity tracking: Tracking user activity, clicks, page views
• Commit logs: Database change logs, replication
• Message queuing: High-throughput message queuing

Basic Example

from kafka import KafkaProducer, KafkaConsumer
import json

# Producer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda v: json.dumps(v).encode('utf-8')
)

producer.send('my-topic', {'key': 'value'})
producer.flush()

# Consumer
consumer = KafkaConsumer(
    'my-topic',
    bootstrap_servers=['localhost:9092'],
    value_deserializer=lambda m: json.loads(m.decode('utf-8')),
    group_id='my-group'
)

for message in consumer:
    print(f"Received: {message.value}")

Common Interview Questions and Answers

1. What is the difference between Kafka and traditional message queues?

Key Difference: Kafka is designed for event streaming and log aggregation, while traditional message queues are designed for task distribution and request-response patterns.

2. Explain the concept of partitions and their importance

Partitions are how Kafka achieves parallelism and scalability:

• Parallelism: Multiple consumers can read from different partitions simultaneously
• Scalability: Add more partitions to increase throughput
• Ordering: Messages within a partition are ordered, but not across partitions
• Replication: Each partition can be replicated across multiple brokers for fault tolerance

Example:

Topic: "user-events"
Partition 0: [msg1, msg2, msg3]
Partition 1: [msg4, msg5, msg6]
Partition 2: [msg7, msg8, msg9]

Key Points:

• Partition count is set at topic creation (can be increased, not decreased)
• Messages with the same key go to the same partition (ensures ordering for that key)
• More partitions = more parallelism but also more overhead

3. How does Kafka ensure message ordering?

Kafka guarantees ordering within a partition, not across partitions:

1. Within a Partition: Messages are strictly ordered

   # All messages with same key go to same partition
   producer.send('topic', key='user-123', value='event1')
   producer.send('topic', key='user-123', value='event2')
   # event1 will always be before event2 for user-123

2. Across Partitions: No ordering guarantee

   • Different partitions can be processed in parallel
   • Messages in different partitions may arrive out of order

3. Ordering Strategies:

   • Key-based: Use same key for related messages
   • Single Partition: Use only one partition (limits parallelism)
   • Application-level: Handle ordering in consumer logic

4. What is consumer group and how does it work?

A consumer group is a set of consumers that work together to consume a topic:

• Load Distribution: Each partition is consumed by only one consumer in the group
• Parallel Processing: Different consumers handle different partitions
• Fault Tolerance: If a consumer fails, its partitions are reassigned to other consumers

Example:

Topic with 3 partitions, Consumer Group with 2 consumers:

Consumer 1 → Partition 0, Partition 1
Consumer 2 → Partition 2

Key Concepts:

• Rebalancing: When consumers join/leave, partitions are reassigned
• Offset Management: Each consumer group maintains its own offset per partition
• Multiple Groups: Different consumer groups can consume the same topic independently

5. Explain KRaft, and what Zookeeper used to do

Zookeeper (Kafka < 4.0) was a separate ensemble that handled:

1. Broker management: tracking which brokers are alive
2. Configuration management: storing topic and broker configuration
3. Leader election: electing partition leaders
4. Consumer group coordination: managing group membership (moved into Kafka itself much earlier, in 0.9)

KRaft (Kafka Raft) replaced it: the controllers form their own Raft quorum and store metadata in an internal Kafka topic, so a cluster is one system instead of two. The practical wins are faster failover, far higher partition counts, and one less thing to operate and secure.

The timeline that matters as of 2026-08:

Interview trap: saying “Kafka uses Zookeeper” or “Zookeeper is being phased out” dates you by several years. On 4.x there is no Zookeeper to phase out. If you’re asked about a migration, the answer is the ZK-to-KRaft migration path, which had to be completed before upgrading to 4.0.

6. How does Kafka handle data retention?

Kafka retains messages based on time and size policies:

1. Time-based Retention:

   log.retention.hours=168  # 7 days
   log.retention.minutes=60

2. Size-based Retention:

   log.retention.bytes=1073741824  # 1 GB per partition

3. Topic-level Configuration:

   kafka-configs --alter --topic my-topic \
     --add-config retention.ms=86400000

Key Points:

• Messages are deleted when retention policy is met
• Retention is per-partition
• Can set retention.ms=-1 for infinite retention
• Old messages are deleted in segments (log segments)

7. What are the different delivery semantics?

Kafka supports three delivery semantics:

1. At-most-once (may lose messages):

   # Producer: fire and forget
   producer.send('topic', value='data')
   
   # Consumer: auto-commit, read and process
   consumer = KafkaConsumer('topic', enable_auto_commit=True)

2. At-least-once (may duplicate messages):

   # Producer: wait for acknowledgment
   future = producer.send('topic', value='data')
   record_metadata = future.get(timeout=10)
   
   # Consumer: manual commit after processing
   consumer = KafkaConsumer('topic', enable_auto_commit=False)
   for message in consumer:
       process(message)
       consumer.commit()

3. Exactly-once (no loss, no duplicates):

   # Requires idempotent producer and transactional consumer
   producer = KafkaProducer(
       bootstrap_servers=['localhost:9092'],
       enable_idempotence=True,
       transactional_id='my-transactional-id'
   )
   
   producer.begin_transaction()
   producer.send('topic', value='data')
   producer.commit_transaction()

8. How to implement exactly-once processing?

Exactly-once requires:

1. Idempotent Producer:

   producer = KafkaProducer(
       bootstrap_servers=['localhost:9092'],
       enable_idempotence=True,  # Prevents duplicates
       acks='all',  # Wait for all replicas
       retries=Integer.MAX_VALUE
   )

2. Transactional Producer:

   producer = KafkaProducer(
       bootstrap_servers=['localhost:9092'],
       transactional_id='unique-id',
       enable_idempotence=True
   )
   
   producer.begin_transaction()
   try:
       producer.send('topic1', value='data1')
       producer.send('topic2', value='data2')
       producer.commit_transaction()
   except Exception:
       producer.abort_transaction()

3. Transactional Consumer:

   consumer = KafkaConsumer(
       'topic',
       bootstrap_servers=['localhost:9092'],
       isolation_level='read_committed'  # Only read committed messages
   )

9. Explain Kafka Streams vs Kafka Connect

Kafka Streams:

• Purpose: Stream processing library for building applications
• Use Case: Transform, aggregate, and process data streams
• Deployment: Embedded in your application
• Example: Real-time analytics, data transformations

from kafka.streams import KafkaStreams

streams = KafkaStreams(builder, config)
streams.start()

Kafka Connect:

• Purpose: Framework for connecting Kafka with external systems
• Use Case: Import/export data to/from Kafka
• Deployment: Standalone or distributed workers
• Example: Connect to databases, file systems, cloud services

# Source connector: Import from database
# Sink connector: Export to database

Key Difference: Streams processes data within Kafka, Connect moves data to/from Kafka.

10. How does Kafka handle backpressure?

Kafka handles backpressure through:

1. Pull-based Model: Consumers pull messages at their own rate

   • Consumers control consumption speed
   • No push from broker to consumer

2. Consumer Configuration:

   consumer = KafkaConsumer(
       'topic',
       fetch_min_bytes=1,  # Minimum bytes to fetch
       fetch_max_wait_ms=500,  # Max wait time
       max_poll_records=500  # Max records per poll
   )

3. Producer Flow Control:

   producer = KafkaProducer(
       bootstrap_servers=['localhost:9092'],
       max_in_flight_requests_per_connection=5,
       buffer_memory=33554432  # 32 MB buffer
   )

4. Partition-level Backpressure: Each partition has its own buffer

   • If one partition is slow, others continue processing
   • Consumers can process fast partitions while slow ones catch up

11. What are the different producer configurations?

Key producer configurations:

1. Acknowledgment (acks):

   # acks=0: Fire and forget (fastest, may lose data)
   # acks=1: Wait for leader acknowledgment (default)
   # acks=all: Wait for all replicas (safest, slowest)
   producer = KafkaProducer(acks='all')

2. Retries:

   producer = KafkaProducer(
       retries=3,  # Number of retries
       retry_backoff_ms=100  # Delay between retries
   )

3. Batching:

   producer = KafkaProducer(
       batch_size=16384,  # Batch size in bytes
       linger_ms=10  # Wait time before sending batch
   )

4. Compression:

   producer = KafkaProducer(
       compression_type='gzip'  # or 'snappy', 'lz4', 'zstd'
   )

5. Idempotence:

   producer = KafkaProducer(
       enable_idempotence=True  # Prevents duplicates
   )

12. How to implement idempotent producers?

Enable idempotence to prevent duplicate messages:

producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    enable_idempotence=True,  # Enables idempotence
    acks='all',  # Required for idempotence
    max_in_flight_requests_per_connection=5,  # Must be <= 5
    retries=Integer.MAX_VALUE  # Required for idempotence
)

# Now duplicate sends are automatically handled
producer.send('topic', key='key', value='value')
producer.send('topic', key='key', value='value')  # Won't create duplicate

How it works:

• Producer assigns sequence numbers to messages
• Broker tracks sequence numbers per producer
• Duplicate sequence numbers are rejected
• Works across producer restarts (uses producer ID)

13. What is the difference between a topic and a partition?

• Topic: Logical category or feed name (e.g., “user-events”, “orders”)
• Partition: Physical division of a topic for parallelism

Analogy: Topic is like a book, partition is like a chapter.

Topic: "user-events"
├── Partition 0 (messages 0-999)
├── Partition 1 (messages 1000-1999)
└── Partition 2 (messages 2000-2999)

Key Points:

• One topic can have multiple partitions
• Partitions enable parallelism and scalability
• Messages are distributed across partitions (by key or round-robin)

14. How does Kafka ensure durability?

Kafka ensures durability through:

1. Replication:

   # Create topic with replication factor 3
   kafka-topics --create --topic my-topic \
     --partitions 3 --replication-factor 3

2. Leader-Follower Model:

   • One partition has a leader (handles reads/writes)
   • Followers replicate data from leader
   • If leader fails, a follower becomes the new leader

3. In-Sync Replicas (ISR):

   • Replicas that are up-to-date with the leader
   • Producer waits for ISR acknowledgment (acks=all)

4. Disk Persistence:

   • Messages are written to disk (not just memory)
   • Uses sequential disk I/O (very fast)

15. What is a consumer offset and how is it managed?

Offset is the position of a consumer in a partition:

• Current Offset: Last message read by consumer
• Committed Offset: Offset saved to Kafka (for recovery)

Offset Management:

1. Auto-commit (at-least-once):

   consumer = KafkaConsumer(
       'topic',
       enable_auto_commit=True,
       auto_commit_interval_ms=5000  # Commit every 5 seconds
   )

2. Manual Commit (at-least-once or exactly-once):

   consumer = KafkaConsumer(
       'topic',
       enable_auto_commit=False
   )
   
   for message in consumer:
       process(message)
       consumer.commit()  # Commit after processing

3. Offset Storage:

   • Stored in __consumer_offsets topic
   • Per consumer group and partition
   • Enables consumers to resume from last position

16. How to handle consumer rebalancing?

Rebalancing occurs when consumers join/leave a group:

Strategies:

1. Cooperative Rebalancing (preferred):

   consumer = KafkaConsumer(
       'topic',
       group_id='my-group',
       partition_assignment_strategy=[
           RoundRobinPartitionAssignor,
           CooperativeStickyAssignor  # Minimizes rebalancing
       ]
   )

2. Handle Rebalance Events:

   from kafka import TopicPartition
   from kafka.consumer.subscription_state import ConsumerRebalanceListener
   
   class MyRebalanceListener(ConsumerRebalanceListener):
       def on_partitions_revoked(self, partitions):
           # Save state, commit offsets
           consumer.commit()
       
       def on_partitions_assigned(self, partitions):
           # Restore state, seek to saved position
           for partition in partitions:
               consumer.seek(partition, saved_offset)
   
   consumer.subscribe(['topic'], listener=MyRebalanceListener())

17. What is the difference between Kafka and RabbitMQ?

Choose Kafka for: High-throughput event streaming, log aggregation, event sourcing Choose RabbitMQ for: Task queues, request-response, complex routing

18. How does Kafka achieve high throughput?

Kafka achieves high throughput through:

1. Sequential Disk I/O: Writes are append-only (very fast)
2. Zero-Copy: Transfers data directly from disk to network
3. Batching: Groups multiple messages together
4. Compression: Reduces network and storage overhead
5. Partitioning: Parallel processing across partitions
6. No Random Disk Access: All reads/writes are sequential

Performance Tips:

• Use batching for producers
• Enable compression
• Tune partition count
• Use appropriate replication factor
• Configure appropriate batch and buffer sizes

19. What is a Kafka broker and how does it work?

A broker is a Kafka server that:

• Stores topic partitions
• Handles producer requests (writes)
• Handles consumer requests (reads)
• Replicates data to other brokers
• Manages leader election

Broker Responsibilities:

1. Request Handling: Process produce/consume requests
2. Partition Management: Manage partition leaders and replicas
3. Metadata Management: Track topics, partitions, and their locations
4. Replication: Replicate data to follower brokers

Cluster Setup:

Broker 1 (ID: 1) → Partitions: topic1-p0 (leader), topic1-p1 (follower)
Broker 2 (ID: 2) → Partitions: topic1-p0 (follower), topic1-p1 (leader)
Broker 3 (ID: 3) → Partitions: topic1-p0 (follower), topic1-p1 (follower)

20. How to monitor Kafka performance?

Monitoring approaches:

1. JMX Metrics:

   • Message rate, throughput
   • Consumer lag
   • Broker metrics

2. Consumer Lag:

   from kafka.admin import KafkaAdminClient
   from kafka.coordinator import ConsumerCoordinator
   
   # Check consumer lag
   admin = KafkaAdminClient(bootstrap_servers=['localhost:9092'])
   # Use tools like Burrow, Kafka Manager, or Confluent Control Center

3. Key Metrics to Monitor:

   • Throughput: Messages per second
   • Latency: End-to-end message latency
   • Consumer Lag: How far behind consumers are
   • Broker CPU/Memory: Resource usage
   • Disk I/O: Read/write performance
   • Network: Bandwidth usage

4. Tools:

   • Kafka Manager: Web UI for cluster management
   • Confluent Control Center: Enterprise monitoring
   • Prometheus + Grafana: Custom dashboards
   • Burrow: Consumer lag monitoring

Interview angle

• “Kafka or a traditional message queue?” - Kafka is a durable, replayable, partitioned log: consumers track their own offset and messages persist after being read. A queue deletes on consumption. Choose Kafka for event streaming, replay and multiple independent consumers; a queue for task distribution.
• “How do partitions relate to ordering and parallelism?” - ordering is guaranteed within a partition only, and one partition is consumed by at most one consumer in a group. So the partition key decides both: same key means same partition means ordered, and partition count caps consumer parallelism.
• “What happens with more consumers than partitions?” - the extras sit idle. Partition count is the ceiling on consumer-group parallelism, and it can be increased but never decreased.
• “Is ZooKeeper still involved?” - no. KRaft replaced it and ZooKeeper was removed entirely in Kafka 4.0 (March 2025). Describing Kafka as depending on ZooKeeper dates you by years.