Branch Prediction and Its Cost in Hot Paths

A mispredicted branch flushes a pipeline of in-flight instructions, and that cost, roughly 15 to 20 cycles, shapes how trading hot paths are written.

Branch prediction and its cost in hot paths matters because a modern processor executes many instructions ahead of the current one speculatively, guessing which way an upcoming conditional branch will go, and when that guess is wrong every speculatively executed instruction has to be discarded and the pipeline refilled from the correct address. A single mispredicted branch on a deep pipeline costs roughly 15 to 20 cycles, and on a hot path that runs millions of times a second, a branch that is unpredictable rather than merely present is often the largest single cost in the function.

A modern core is pipelined into many stages, fetching, decoding and issuing instructions well before earlier instructions have finished executing, and it relies on a branch predictor, a small piece of hardware that tracks the recent history of a given branch and guesses which way it will go next time. When the guess is right, the pipeline never stalls and the branch is effectively free. When the guess is wrong, everything the processor speculatively executed past that branch has to be thrown away and refetched, and that flush is where the 15 to 20 cycle cost comes from.

The predictor is good at branches with a consistent pattern, a loop that always runs the same number of iterations, for instance, and poor at branches whose outcome genuinely depends on data the processor cannot know in advance, which describes most of the conditionals in trading code, since they are testing live market data rather than a fixed loop bound.

A condition like whether an incoming price improves the best bid is effectively random from the processor's point of view across a long enough sequence of messages, since real order flow does not follow a pattern the branch predictor's history table can exploit. Code that structures its logic around several such conditions in sequence, each one dependent on live data, accumulates misprediction cost on nearly every message, which is why low latency code is often restructured specifically to reduce how many data-dependent branches sit on the path a message travels.

One common technique replaces an if statement with a conditional move or with bit arithmetic that computes both outcomes and selects between them without a branch at all, trading a small amount of extra computation, which the processor can pipeline freely, for the elimination of an unpredictable branch, which it cannot. C++20 also added the likely and unlikely attributes, which let a programmer tell the compiler which side of a branch is expected to execute more often, helping the compiler lay out the generated code so the common path stays contiguous in instruction memory.