10,000 clients hold a cache entry that expires at the same instant, each retries a failed backend call after exactly 1 second and doubles on each failure, and the backend serves 2,000 requests per…

Question

10,000 clients hold a cache entry that expires at the same instant, each retries a failed backend call after exactly 1 second and doubles on each failure, and the backend serves 2,000 requests per second. Explain why the system never recovers, compute the arrival rate under full jitter over 10 seconds, and give the retry policy you would set.

Accepted Answer

Synchronised backoff preserves the herd, so the backend keeps receiving 10,000 arrivals in one burst against a capacity of 2,000 per second and never drains, while full jitter spread over 10 seconds gives 10,000 / 10 = 1,000 per second, inside capacity, so the policy is exponential backoff with full jitter, a cap, and a retry budget. Doubling the delay without randomising it changes when the herd arrives and never breaks it apart, because every client doubles at the same instant from the same starting point: round two is 10,000 requests at t = 1, round three is 10,000 three seconds in, and each round overloads, fails, and re-forms the herd. That is the thundering herd, and adding capacity does not solve it because the peak scales with the client count. Full jitter draws the delay uniformly from 0 to the current backoff instead of using the backoff itself, so arrivals become uniform across that interval and the queue drains. Add two controls. A retry budget caps retries at a fraction of successful traffic, around 10 percent, so an outage cannot be amplified into a self inflicted denial of service. And request coalescing on the cache key lets one client refill the entry while the other 9,999 wait on that single call, which removes the herd at its source.