Measuring Latency Properly in a Trading System

The mean hides the tail that actually costs money, so trading systems are graded on the 99th and 99.9th percentile rather than average latency.

Measuring latency properly in a trading system means reporting the 99th and 99.9th percentile rather than the mean, because the mean is actively misleading for a system whose cost is driven by its worst moments rather than its typical ones. A system that responds in 200 nanoseconds nearly every time but occasionally spikes to 50 microseconds can have a mean latency that looks excellent while still being picked off repeatedly during the rare moments its response is slow, and averaging across millions of messages hides exactly those moments from view.

Latency in a real trading system is not distributed the way a classroom example assumes. A garbage collection pause in a managed runtime, an operating system scheduling a competing process onto the same core, a cache line evicted by an unrelated thread, or a page fault on a rarely touched code path can each independently produce a large, rare spike, and the combined distribution has a long right tail even when the typical case is fast. A single large outlier pulls the mean upward without revealing how often outliers of that size actually occur, while a percentile figure states directly what fraction of messages exceeded a given threshold.

This matters commercially because the cost of a slow response is not linear in its size. A market maker whose quote is stale by 50 microseconds during a fast market move can be picked off by a faster competitor for the full size of the quote, so the handful of worst messages in a distribution can account for a disproportionate share of total losses even though they represent a tiny fraction of total message volume.

A 99th percentile latency figure states that 99 percent of messages completed at or below that value, meaning one message in a hundred was slower, and the 99.9th percentile narrows that to one message in a thousand. At the message volumes a firm like Optiver processes across a single trading session, one message in a thousand still represents thousands of individual events, so the tail is not a rounding error, it is a population large enough that its behavior has to be engineered deliberately rather than accepted as noise.

Some systems go further and track the 99.99th percentile or the absolute maximum observed, since a kill switch or a risk check firing late even once can matter regardless of how rarely it happens, and a distribution's tail beyond the 99.9th percentile often behaves differently from the rest of the curve entirely, driven by qualitatively different causes such as an operating system jitter event rather than ordinary cache variance.