Apache Kafka
Apache Kafka is a distributed streaming platform for building real-time data pipelines and streaming applications.
Key Concepts
Core Components
- Producer: Applications that publish messages to topics
- Consumer: Applications that read messages from topics
- Topic: Category or feed name to which messages are published
- Partition: Topics are divided into partitions for parallelism
- Broker: Kafka server that stores topic data
- Cluster: Group of brokers working together
- Zookeeper: Coordinates the cluster (being phased out in newer versions)
Architecture
- Distributed: Runs as a cluster of servers
- Fault-tolerant: Replicates data across multiple brokers
- Scalable: Can handle millions of messages per second
- Real-time: Processes streams of records
Common Interview Questions
- What is the difference between Kafka and traditional message queues?
- Explain the concept of partitions and their importance
- How does Kafka ensure message ordering?
- What is consumer group and how does it work?
- Explain the role of Zookeeper in Kafka
- How does Kafka handle data retention?
- What are the different delivery semantics?
- How to implement exactly-once processing?
- Explain Kafka Streams vs Kafka Connect
- How does Kafka handle backpressure?
- What are the different producer configurations?
- How to implement idempotent producers?