Amazon Kinesis

AWS’s streaming data platform. The two pieces that matter: Kinesis Data Streams (ordered, replayable stream — the Kafka-like one) and Kinesis Data Firehose (managed buffered delivery to S3/Redshift/OpenSearch). For a backend role the interview centers on the shard model and the “vs SQS/Kafka/EventBridge” comparison.

Kinesis Data Streams — the shard model

A stream is made of shards. Each shard has fixed capacity:

• Ingest: 1 MB/s or 1,000 records/s.
• Egress: 2 MB/s (shared across all standard consumers of that shard).

Stream throughput = shard count × per-shard capacity. To scale, you add shards (resharding). Records are distributed to shards by a partition key — hash(partition_key) picks the shard.

producer → partition key → hash → shard 1 ─┐
                                  shard 2 ─┼→ consumers
                                  shard 3 ─┘

Ordering is per-shard. All records with the same partition key go to the same shard and are delivered in order. There’s no global ordering across shards — same as Kafka partitions.

Hot shard problem: if one partition key is very popular (or you use too few distinct keys), all that traffic lands on one shard and you hit the per-shard limit while other shards sit idle. Same failure mode as a DynamoDB hot partition. Fix: a higher-cardinality partition key.

Retention and replay

Records stay in the stream for a retention period — default 24 hours, configurable up to 365 days. Within that window, consumers can re-read from any point: by sequence number, by timestamp, or TRIM_HORIZON (oldest available) / LATEST.

This is the key difference from SQS: a Kinesis record isn’t consumed-and-gone. Multiple independent consumers each track their own position, and you can replay history (reprocess after a bug, backfill a new consumer).

Consumers — three ways

For most backend work: Lambda ESM (simple) or KCL (a dedicated consumer app). Enhanced fan-out when you have several consumers competing for the same shard’s egress.

Lambda + Kinesis gotchas

• Batches are per-shard and ordered — a failing batch blocks that shard until it succeeds or is dropped. Use BisectBatchOnFunctionError and a failure destination / maxRecordAge so one poison record doesn’t stall the shard forever.
• Lambda concurrency for a Kinesis source is per shard (with ParallelizationFactor to run multiple invocations per shard).
• At-least-once: your processing must be idempotent.

Capacity modes

• Provisioned — you set the shard count; you pay per shard-hour; you manage resharding.
• On-demand — Kinesis scales shards automatically with traffic; you pay per GB ingested/retrieved. Easier; pricier per GB at steady high volume.

Same trade-off shape as DynamoDB’s on-demand vs provisioned: on-demand for spiky/unknown, provisioned for steady and right-sized.

Kinesis Data Firehose

Not a stream you consume — a managed delivery pipeline. You put records in; Firehose buffers them (by size or time) and delivers to a destination: S3, Redshift, OpenSearch, Splunk, or an HTTP endpoint. It can transform records in-flight with a Lambda and convert formats (JSON → Parquet).

If the goal is just “land streaming data in S3 as Parquet,” Firehose — no consumer to write. If you need to process the stream in real time with custom logic and replay, Data Streams.

Kinesis vs SQS vs EventBridge vs Kafka

Decision shortcut:

• Work queue, one consumer per message → SQS.
• Route events to many targets with filtering → EventBridge.
• High-throughput ordered stream, multiple independent consumers, replay → Kinesis Data Streams.
• Same as Kinesis but you need Kafka APIs/tooling or extreme throughput → MSK (managed Kafka).
• Just deliver streaming data into S3/Redshift/OpenSearch, no consumer code → Firehose.

Common gotchas

• Hot shard — low-cardinality partition key sends everything to one shard; other shards idle while you throttle.
• Per-shard egress is shared — N standard consumers split 2 MB/s; use enhanced fan-out when consumers compete.
• Lambda batch failure blocks the shard — ordered processing means a poison record stalls the shard; configure bisect-on-error + failure destination + maxRecordAge.
• Resharding isn’t instant — scaling provisioned streams takes time; on-demand avoids the management but costs more per GB.
• Confusing Firehose with a stream — Firehose has no replay, no ordering, no consumers; it’s a delivery pipe.
• Retention costs — longer retention windows cost more; default 24h is often enough.

Interview angle

• “How does Kinesis ordering work?” — per-shard. The partition key hashes to a shard; all records with that key go to that shard in order. No global ordering across shards — same model as Kafka partitions.
• “What’s the hot shard problem?” — a low-cardinality or skewed partition key sends disproportionate traffic to one shard, which throttles at 1 MB/s while other shards sit idle. Fix: higher-cardinality partition key.
• “Kinesis vs SQS?” — SQS is a queue: a message is consumed by one consumer and gone, no replay. Kinesis is a retained stream: multiple independent consumers each track their own position, and you can replay within the retention window. Use SQS for work distribution; Kinesis for streaming with multiple readers and replay.
• “Kinesis vs Kafka (MSK)?” — same conceptual model (sharded/partitioned ordered streams with retention). Kinesis is more managed (no cluster), shard-based capacity. MSK is managed Kafka — pick it when you need Kafka APIs, the Kafka ecosystem/tooling, or higher throughput.
• “Data Streams vs Firehose?” — Data Streams is a stream you write consumers for, with replay and per-shard ordering. Firehose is a managed buffered delivery pipe to S3/Redshift/OpenSearch — no consumers, no replay, no ordering. If you just need streaming data landed in S3, Firehose.
• “How do you process a Kinesis stream with Lambda safely?” — Lambda event source mapping polls per shard. Because processing is ordered, a failing batch blocks that shard — so configure BisectBatchOnFunctionError, a failure destination, and maxRecordAge so a poison record doesn’t stall the shard. Make processing idempotent (at-least-once).