Cache Locality and Why It Dominates Trading Code Performance

An L1 hit costs about 1 nanosecond and a main memory access costs 80 to 100, and that gap is why cache locality outweighs algorithmic cleverness.

Cache locality dominates trading code performance because the gap between a cache hit and a main memory access is enormous relative to the cost of the arithmetic the code is actually trying to perform. An L1 cache hit costs around 1 nanosecond, an L2 hit costs around 4 nanoseconds, and a main memory access costs 80 to 100 nanoseconds, so a single pointer chase that misses every level of cache costs more than a hundred ordinary add or compare instructions. Almost every design choice in a low latency hot path, from container selection to struct layout, exists to keep the data the processor needs sitting in cache rather than sending it back out to memory.

A modern processor core runs at a few billion cycles per second, and an L1 cache access typically completes in three to four cycles because the cache sits physically adjacent to the execution units and holds only tens of kilobytes. Main memory sits across a bus that has to arbitrate with other cores, and a request that misses the last level of cache can take two to three hundred cycles round trip. The processor is not idle during that wait in the strict sense, since modern cores can execute other instructions out of order, but a hot path that repeatedly stalls waiting on memory cannot hide that latency no matter how the rest of the code is written.

This is why a candidate preparing for a firm interview is asked to reason in terms of cache misses rather than instruction counts. Two implementations of the same function can have identical algorithmic complexity and differ by an order of magnitude in wall clock time purely because one keeps its working set inside a 32 kilobyte L1 cache and the other scatters it across memory.

Data moves between memory and cache in fixed 64 byte blocks called cache lines, so the layout of a struct determines how many of those blocks a given operation touches. A struct with fields ordered from largest to smallest wastes less space to padding than one with fields declared in an arbitrary order, and a struct that groups the fields read on the hot path together, separate from fields that are only read occasionally, means a single cache line load serves many consecutive operations instead of one.

This second technique, often called hot and cold splitting, matters more as a struct grows. An order object that stores a price and a quantity next to a rarely read audit timestamp forces every hot path read to pull in the timestamp's cache line whether it needs it or not, and moving the audit fields into a separate structure entirely removes that cost.