Kafka Exactly-Once: At-Least-Once Plus Dedup

“Exactly-once” sounds impossible on an unreliable network. What does Kafka actually guarantee, and how?

Exactly-once never means delivered once — it's at-least-once delivery plus dedup so the effect is once; Kafka does it in two layers: an idempotent producer (retry dedup) and transactions (atomic read-process-write).

The myth-buster

A producer sends a message, the broker writes it, but the acknowledgment is lost on the way back. The producer, hearing nothing, can’t know whether the broker got it. So it either retries (risking a duplicate) or doesn’t (risking a loss). There is no third option at the delivery layer. Exactly-once is therefore never about delivering once; it’s about making duplicates invisible: deliver at-least-once, then deduplicate so the observable effect is as if it happened once.

Layer 1 — the idempotent producer (kills retry duplicates)

Give each write an identity and let the destination reject repeats. With idempotence enabled, the producer gets a producer id (PID) and stamps every message with the PID plus a monotonic sequence number per partition. The broker remembers the last sequence it accepted for each (PID, partition); when a retry arrives carrying a sequence it has already seen, it acknowledges but does not append again. Same “stamp it with an id, reject the repeat” move as LSM tombstones and idempotent consumers. It’s cheap enough to be on by default — but it only covers one producer’s retries to a partition.

Layer 2 — transactions (atomic read-process-write)

The real pipeline consumes from topic A, produces results to topic B, then commits its offset on A. That’s two writes that must happen as a unit. If the app produces to B and crashes before committing the offset, it restarts, reprocesses the same input, and produces to B again — a duplicate the idempotent producer can’t stop, because these are genuinely new writes.

So Kafka makes produce-and-commit-offset a transaction. A producer with a stable transactional.id begins a transaction, writes to B, writes the consumer offsets (the offset commit is itself just a write to __consumer_offsets, so it rides along), then commits once. A transaction coordinator writes begin/commit markers — a two-phase-commit-style protocol.

Why the read side matters: consumers set isolation.level=read_committed and won't deliver a transaction's messages until they see its commit marker. Uncommitted/aborted messages are physically in the log but invisible — exactly the MVCC visibility rule: written, but a reader sees it only if the writer committed. And transactional.id fences a restarted "zombie" producer, the single-writer idea from Raft leader election.

⚠ Common misconception: "exactly-once covers my whole app." It's airtight only inside Kafka. The moment processing has an external side effect — charging a card, sending an email, writing to another database — that's outside the transaction, and you must supply idempotency there yourself (dedup keys, upserts).

Recall checkpoint

1. Why is "exactly-once = delivered once" a myth? A lost ack forces the producer to retry-or-lose on an unreliable network; you can't deliver exactly one time, only dedup so the effect is once.

2. What makes uncommitted transactional messages invisible to readers? A read_committed consumer won't surface them until the transaction's commit marker appears — MVCC visibility applied to a log.

TL;DR

Exactly-once = at-least-once + dedup; delivering exactly one time is impossible (lost acks force retry-or-lose).
Idempotent producer: PID + per-partition sequence number → broker absorbs retries → kills retry duplicates (on by default).
Transactions: a stable transactional.id wraps output writes + offset commit atomically via a coordinator’s begin/commit markers.
read_committed hides uncommitted messages until the commit marker — MVCC visibility on a log; transactional.id fences zombie producers.
Guarantee is airtight only inside Kafka; external side effects need their own idempotency.