Everything about storage engines lived on a single machine, where the enemy was disk I/O. Distributed data changes the enemy to the network — and on a real network, cut cables, dropped packets, and GC pauses guarantee it will fail.
The scenario
You keep two copies of the data on two servers, A and B, replicas of each other. A write updates A. A microsecond later — before A has told B — the cable between them is cut. Now a read for that just-written key is routed to B, which can’t reach A. A client is waiting. B has exactly two options, and they’re mutually exclusive:
- Answer from what it has. B stays responsive but may return stale data (or a 404 for a key that now exists on A). This is choosing Availability and sacrificing Consistency — an AP system.
- Refuse until it can confirm it’s current. Every answer is correct, but while the partition lasts B is effectively down for that request. This is choosing Consistency and sacrificing Availability — a CP system.
“Just reroute B’s request to A” feels like an escape, but it isn’t: rerouting only works if B can reach A, and the whole premise is that it can’t. Rerouting is a normal-operation optimization, not a partition strategy.
The theorem, stated properly
When a Partition happens, you can preserve Consistency or Availability, but not both. The single most important correction to how CAP is usually taught: it is not “pick 2 of 3.” Partition-tolerance isn’t optional — you don’t get to choose whether cables get cut; physics chooses for you. So P is always required, and the only real decision is C or A during a partition. A “CA” system doesn’t exist in the real world; a single-node database is “CA” only in the trivial sense that it has no network to partition. The genuine choice is CP vs AP.
It maps onto systems you know
- CP — a bank ledger or config store that must never disagree with itself would rather reject a request than return a wrong balance. This is why etcd, ZooKeeper, and Postgres with synchronous replication refuse writes when they can’t reach a quorum.
- AP — a shopping cart, a social feed, or DNS would rather show slightly stale data than nothing, and reconcile the divergence afterward.
There’s no universally right answer, only right-for-this-workload — the same “the workload decides” instinct from MongoDB modeling, now at the level of the whole system. And CAP only covers the partition case; what happens the rest of the time is PACELC.