A benchmark harness sends a request every 10 microseconds, measures the response, and reports a 99th percentile of 30 microseconds. The system under test stalls for 10 milliseconds once per second. Explain coordinated omission here and compute what the corrected 99th percentile is.

A benchmark harness sends a request every 10 microseconds, measures the response, and reports a 99th percentile of 30 microseconds. The system under test stalls for 10 milliseconds once per second. Explain coordinated omission here and compute what the corrected 99th percentile is.

Approach: Ask what the harness does during the stall, count how many requests should have been sent in that window, and rebuild the latency distribution including the requests that were never issued.

The corrected 99th percentile is about 5 milliseconds, and the 30 microsecond figure is coordinated omission: the harness stopped sending during the stall. A closed-loop harness that waits for each response before sending the next issues nothing during the 10 ms stall, so instead of 1000 delayed samples it records one sample of 10 ms and then resumes. Per second the harness intends 100,000 requests, and the stall should produce 1000 of them with latencies uniformly spread from 10 ms down to zero as the queue drains, averaging 5 ms. Those 1000 samples are 1% of the second, so they land exactly at the 99th percentile and above, putting the true 99th percentile around 5 ms rather than 30 microseconds, and the 99.9th around 9 ms. The error is a factor of 160. The fixes are to run open loop, sending on a fixed schedule regardless of outstanding responses, or to correct after the fact by backfilling synthetic samples for the intended send times that were skipped. Any harness that applies backpressure to itself measures the system's good behaviour and hides its bad behaviour, which is the opposite of what a desk needs.

Follow-up: Open loop generation can push the system past its service rate and make the queue grow without bound. How do you distinguish that artifact from a genuine tail?

Key concepts: coordinated omission, percentile, tail latency, backpressure.