backend / message queues / 00_broker_comparison.md

Kafka vs RabbitMQ — choosing a message broker

5 min read source

Kafka vs RabbitMQ — choosing a message broker

The most common messaging interview question. The wrong answer is “Kafka is faster.” The right answer starts from the model: RabbitMQ is a smart broker with dumb consumers (broker routes, tracks, and deletes messages); Kafka is a dumb broker with smart consumers (broker is an append-only log, consumers track their own position).

Deep dives: rabbitmq/01_what_is_rabbitmq.md, kafka/01_what_is_kafka.md.

The core difference: queue vs log

RabbitMQ (queue):                    Kafka (log):
producer → exchange → queue          producer → topic partition
consumer ACKs → message DELETED      consumer reads at offset → log UNCHANGED
                                     another consumer group re-reads from 0
  • RabbitMQ: a message is consumed destructively. Once acked, it’s gone. Competing consumers on one queue split the work.
  • Kafka: messages are retained for a configured time/size regardless of consumption. Consumers are just cursors (offsets) over the log. New consumers can replay history.

Everything else follows from this.

Side by side

Dimension RabbitMQ Kafka
Model message queue (AMQP) distributed, partitioned log
Consumption destructive, per-message ack offset-based, non-destructive
Replay no (message deleted on ack) yes — rewind offset, add new group
Routing rich: direct/topic/fanout/headers exchanges none in broker — topic + partition key only
Ordering per-queue (breaks with multiple consumers/requeues) strict per-partition
Delivery tracking broker tracks each message’s state broker tracks nothing; consumers commit offsets
Per-message features TTL, priorities, delayed delivery, DLX none (build in consumer or with retry topics)
Retention until consumed (or TTL) time/size-based (or compacted), days–forever
Fan-out to N consumers bind N queues to one exchange N consumer groups read same log — free
Throughput ceiling tens of thousands msg/s per node (typical) millions msg/s per cluster (sequential disk I/O, batching, zero-copy)
Latency very low (sub-ms possible), push-based low but batch-oriented, pull-based
Consumer scaling add consumers to a queue, instant bounded by partition count; triggers rebalance
Protocol AMQP 0-9-1 (+ MQTT, STOMP) custom binary protocol
Typical Python client pika, aio-pika confluent-kafka, aiokafka

When RabbitMQ fits

  • Task distribution / background jobs — work queues with fair dispatch, retries, DLX. This is why Celery uses it (celery/01_what_is_celery.md).
  • Complex routing — route by routing-key patterns or headers without consumer-side filtering (rabbitmq/02_rabbitmq_exchanges.md).
  • Per-message control — priorities, per-message TTL, delayed retries, dead-lettering (rabbitmq/03_exchanges_queues_dlx.md).
  • RPC over messaging — request/reply with reply_to + correlation_id.
  • Low-volume, low-latency command passing between services where each message is an instruction to do work once.

When Kafka fits

Rule of thumb: commands → RabbitMQ, events → Kafka. A command (“charge this card”) should be executed once by one worker and disappear. An event (“order placed”) is a fact that many consumers, present and future, may care about.

The rest of the field

Broker One-liner Reach for it when
SQS / SNS managed queue / pub-sub on AWS you’re on AWS and want zero ops; no replay, 14-day max retention
Redis Streams log-like structure inside Redis you already run Redis and need lightweight streaming; not a durability story
NATS (JetStream) lightweight cloud-native messaging ultra-low latency service mesh–style messaging, simpler ops than Kafka
Pulsar Kafka competitor, segmented storage multi-tenancy, tiered storage, queue+stream in one system
IBM MQ enterprise legacy standard you didn’t choose it; the bank did (ibm_mq/01_ibm_mq_overview.md)

Note Celery’s broker support: RabbitMQ and Redis are first-class; Kafka is not supported. If the team is Celery-based, “just use Kafka for tasks” is not a drop-in swap.

Using both is normal

A common production shape:

  • RabbitMQ (or SQS) for work queues: emails, PDF generation, payment execution.
  • Kafka for the event backbone: order/user/payment events feeding search indexing, analytics, notifications, and the data warehouse.

The saga/outbox machinery (../14_microservices/04_data_consistency_patterns.md) works over either; choose per stream of data, not per company.

Common interview confusions

  • “Kafka is a message queue.” A consumer group makes it behave queue-like (each message processed by one member), but there’s no per-message ack/delete, no broker-side retry, no DLQ primitive — you build those with retry topics and consumer logic (kafka/04_delivery_semantics.md).
  • “RabbitMQ doesn’t scale.” It scales to very high volumes with clustering and quorum queues; it just doesn’t scale the same way (no partition-parallel consumption model, ordering vs parallelism trade-offs hit earlier).
  • “Kafka guarantees ordering.” Only per partition. Order across a topic requires one partition (killing parallelism) or a partition key that groups what must be ordered (e.g., order_id).
  • “Exactly-once means I can forget idempotency.” No — see kafka/04_delivery_semantics.md; end-to-end you still design idempotent consumers (celery/04_idempotency.md).

Interview angle

  • “Kafka vs RabbitMQ — when would you pick each?” — Lead with queue-vs-log, then: destructive consumption vs replay, broker routing vs partition keys, commands vs events. Give one concrete example of each.
  • “How would you add a new consumer of order events a year after launch?” — Kafka: new consumer group, replay from offset 0 (if retention allows). RabbitMQ: you can’t — messages are gone; you’d need to re-emit or backfill from the DB.
  • “Can you get strict global ordering in Kafka?” — One partition only; explain the throughput cost and the partition-key compromise.
  • “Your team uses Celery — where does Kafka fit?” — Not as the Celery broker; as the event backbone next to it.