Kafka Partitions and Replication: Where Availability Is Spent

One log lives on one machine, capped by one disk. How does Kafka scale it — and what guarantee do you lose the instant you split it?

A topic splits into partitions — the unit of parallelism, ordering scope, and distribution; a partition key keeps each entity's events ordered within one partition but sacrifices global order, and durability-vs-availability is set by acks + min.insync.replicas.

The partition key routes ordering

When a producer sends a message it can attach a key — user_id, order_id, whatever’s order you care about. Kafka computes partition = hash(key) % numPartitions, so the same key always lands in the same partition, in produced order. Keyless messages spread round-robin for balance. Rule: key by the entity whose ordering must be preserved. Two different orders may sit in different partitions with no ordering relationship — which is fine, because you never cared about order between unrelated orders.

A partition is the unit of three things

Parallelism (within a consumer group, one partition is consumed by exactly one consumer — so partition count caps consumer parallelism), ordering scope (order holds only within a partition; offsets are unique per-partition, not per-topic), and distribution (partitions spread across brokers).

⚠ Common misconception: "Partition count = number of message types." No — partition count is your parallelism / throughput dial. A single-type topic keyed by user_id might have 50 partitions for 50-way parallelism. Too few caps throughput; too many costs failover time, file handles, and latency.

What you lose by splitting: global order. A single log gives one total order for free. Split into partitions and there's no cross-partition order — offset 5 in partition 0 is unrelated to offset 5 in partition 2. A true global total order requires a single partition, which throws away all parallelism. That's the core Kafka tradeoff.

Replication: a broker dying is not a wait

Each partition is replicated across brokers (replication factor ~3): one leader and several followers. All reads and writes go through the leader; followers continuously copy the leader’s log (shipping the WAL over the network). The followers currently caught up form the ISR (in-sync replica set). When the leader dies, the controller elects a new leader from the ISR and clients redirect — the partition keeps serving. It does not wait for the dead machine.

acks: the durability-vs-availability dial

This is Kafka’s version of the quorum-commit tradeoff:

`acks`	Leader acknowledges after…	On leader crash	Trades for
`0`	not waiting at all	message silently lost	max throughput
`1`	it writes locally, before followers copy	✓lost if no follower had it	speed
`all`	all in-sync replicas have it	✓survives — any new leader has it	durability

acks=all with min.insync.replicas (e.g. 2 of 3) is the CP choice: if too few replicas are in sync, the leader refuses the write rather than accept one it can’t safely replicate — the same “reject rather than risk losing an acked write” from CAP/consensus.

Recall checkpoint

1. What guarantee do you lose by splitting a topic into partitions? Global ordering — order is preserved only within a partition, and offsets are unique per-partition, not per-topic.

2. What does acks=all guarantee, and what does it cost? A committed message survives any leader failure (every ISR member already has it), at the cost of higher latency and reduced availability when too few replicas are in sync.

TL;DR

A topic splits into partitions = the unit of parallelism, ordering scope, and distribution.
A partition key (hash(key) % numPartitions) keeps each entity’s events ordered in one partition.
Splitting sacrifices global order; offsets are per-partition, not per-topic; total order needs one partition (no scale).
Each partition is replicated (leader + ISR); a broker death triggers a fast leader election, not a wait.
acks + min.insync.replicas set durability vs availability; acks=all is the CP quorum-commit choice.