Kafka Is a Log, Not a Queue

A traditional queue deletes a message the moment it’s read. Kafka keeps it. Why — and what does that buy you?

A Kafka partition is an append-only, immutable log; a consumer never removes anything — it just holds an offset (a cursor into the log) and advances it — so many consumers read the same stream independently, and you can replay history.

Say your app emits order events, and three services care: a fraud checker, an analytics pipeline, and an email sender.

The inversion

A traditional message queue is destructive: a consumer takes a message, it’s popped off, and each message goes to exactly one consumer. Kafka does the opposite. A topic partition is an append-only, immutable, ordered log on disk — literally the write-ahead log from the storage-engine world, which is why it’s fast: it’s the sequential-I/O win.

Producers append messages at the tail.
Each message gets a monotonically increasing offset — its position in the log.
Messages are retained for a configured time regardless of whether anyone read them.
A consumer holds an offset and advances it as it reads. It deletes nothing.

Two consumers read the same log at their own offsets; neither erases anything.

The superpower

Because the log persists and every consumer carries its own cursor, you get what a destructive queue can’t: many independent consumers each read the whole stream at their own pace (fraud, analytics, and email all consume the same order events, independently), and you can replay — a new or recovering service resets its offset to 0 and re-reads everything still retained. The log is the durable source of truth; consumers are cheap, disposable cursors over it.

⚠ Common misconception: "Reading a Kafka message consumes it, like a queue." No — reading removes nothing. Retention is set by config (time/size), not by consumption, and each consumer's position is its own offset, independent of every other consumer.

Recall checkpoint

1. What does a consumer store instead of deleting messages? Its own offset — a cursor pointing at its position in the log.

2. How does a service replay history? It resets its offset to 0 (or any earlier point) and re-reads everything still within the retention window.

TL;DR

A Kafka partition is an append-only, immutable log — not a destructive queue.
Each message gets a monotonic offset; consumers advance their own offset and delete nothing.
Messages are retained by config, regardless of whether they were read.
Superpower: many independent consumers + replay by resetting the offset.
It’s fast because it’s a sequential-I/O append log — the WAL, again.