Caching: Cache-Aside and the Thundering Herd

A cache is supposed to protect your database. How can it end up being the thing that takes the database down?

A cache trades staleness for speed and load relief; but when many requests miss at once — a hot key expires, or the cache goes cold — they all fall through to the database together, a thundering herd that can crush the very database the cache was shielding.

A read-heavy service puts Redis in front of Postgres. The point of the cache isn’t durability — it’s to cut latency (memory vs disk + network) and to cut load (keep repeated reads off the database).

The cache-aside pattern

The most common pattern is cache-aside (lazy loading), and the read path is:

Look in the cache. Hit → return it.
Miss → read from the database, write the value into the cache, then return it.

Writes update the database and invalidate (or update) the cached key. The cache only ever holds what’s been asked for recently, and it stays roughly in sync because stale keys are evicted or overwritten.

TTL and eviction

Two mechanisms bound the cache:

TTL (time to live) — each key expires after a set time, bounding how stale a read can be.
Eviction — memory is finite, so when full the cache drops keys, usually LRU (least-recently-used): the coldest key goes first, keeping the hot working set resident.

The failure mode: thundering herd

Now the trap. Suppose one very hot key — a popular product, the homepage feed — is requested thousands of times a second, and its TTL expires. In the instant after expiry, every one of those concurrent requests misses, and every one falls through to the database to recompute the same value. One expiry becomes thousands of simultaneous identical queries.

⚠ Common misconception: "a cache always reduces database load." Only in steady state. A mass miss — a hot key expiring, or a cold/restarted cache, or a cluster resize (see consistent hashing) — inverts it: the cache concentrates load into a spike the database never sees under normal traffic. This is the cache stampede / dogpile.

Taming the herd

Request coalescing / single-flight — let only the first misser recompute the value; everyone else waits for that one result. Thousands of misses become one query.
Probabilistic early expiration — let a key refresh slightly before its TTL, at random, so it’s renewed by one request while still serving the rest — the herd never forms.
Stale-while-revalidate — serve the stale value immediately and refresh in the background.

Recall checkpoint

1. In cache-aside, what happens on a miss? Read from the database, write the value into the cache, then return it.

2. Why can a single hot key expiring hurt more than a cold key expiring? A hot key has many concurrent readers, so its expiry produces many simultaneous misses that stampede the database at once.

TL;DR

A cache trades staleness for latency + load relief — it is not a source of truth.
Cache-aside: on miss, read the DB, populate the cache, return; writes invalidate the key.
TTL bounds staleness; LRU eviction keeps the hot set when memory is full.
Thundering herd / stampede: a mass miss (hot-key expiry, cold cache, resize) sends everyone to the DB at once.
Fixes: single-flight / request coalescing, probabilistic early expiration, stale-while-revalidate.