A consumer in your group dies — or just gets slow. How does Kafka notice, who takes its partitions, and do you lose messages or process them twice?
Consumers heartbeat a group coordinator; silence past a timeout triggers a rebalance that reassigns partitions, and the inheritor resumes from the last committed offset — so whether you skip or duplicate depends on committing before or after processing.
A group of 3 consumers shares a 6-partition topic (each partition goes to exactly one consumer in the group — that’s parallelism without breaking per-partition order). Consumer #2 stops.
How the group even notices
Every consumer sends periodic heartbeats to a broker acting as the group coordinator. When #2’s heartbeats stop for longer than session.timeout.ms, the coordinator declares it dead and triggers a rebalance.
poll() — e.g. processing a batch takes longer than max.poll.interval.ms. Slow processing, not crashes, is the number-one cause of rebalance storms.The offset decides duplicate vs skip
The inheritor resumes from the last committed offset for the partition — offsets are committed periodically into the internal __consumer_offsets topic, not on every message. So the recovery point is “where #2 last committed,” not “where #2 actually was.” That gap is the whole fork, and a crash between processing and committing is unavoidable — so you choose which failure you can tolerate:
| Order | Crash in the gap means… | Semantics |
|---|---|---|
| Commit, then process | offset says done but work never ran → inheritor skips it | at-most-once (loss) |
| Process, then commit | work ran but offset not saved → inheritor reprocesses it | at-least-once (duplicate) |
The industry default is at-least-once (process then commit) plus idempotent processing — handling the same message twice has the same effect as once (upserts keyed by message id, dedup tables). Duplicates are survivable; silent loss usually isn’t.
Rebalancing hell
The classic pain is stop-the-world rebalancing: in the old protocol, any join or leave revokes every partition from every consumer and reassigns from scratch, so the whole group stops. Chain a few consumers flapping on max.poll.interval.ms under load and slowness causes rebalances that cause more slowness. Fixes: cooperative/incremental rebalancing (move only the partitions that must move) and static membership (a restarting consumer reclaims its same partitions, so rolling deploys don’t thrash).
Recall checkpoint
1. Why might a group rebalance when nothing crashed? A live consumer whose processing exceeds max.poll.interval.ms stops calling poll() and is evicted.
2. Commit-then-process vs process-then-commit — which risks duplicates, which risks loss? Commit-first risks loss (at-most-once); process-first risks duplicates (at-least-once).
TL;DR
- Each partition → exactly one consumer per group; same group = work-sharing, different groups = fan-out.
- Failure detection = heartbeats to the coordinator; silence past
session.timeout.ms(or missingpoll()pastmax.poll.interval.ms) → eviction → rebalance. - The inheritor resumes from the last committed offset (
__consumer_offsets), not the dead consumer’s true position. - Commit-then-process → skip/loss (at-most-once); process-then-commit → duplicate (at-least-once).
- Default: at-least-once + idempotent processing. “Rebalancing hell” usually = slow processing, fixed by cooperative rebalancing + static membership.