A pricing loop reads 8 million market data updates per second, each touching one 64-byte cache line chosen at random from a 2 GB book. The box has 200 GB/s of memory bandwidth. Show why bandwidth is not the binding constraint, then say what is.

A pricing loop reads 8 million market data updates per second, each touching one 64-byte cache line chosen at random from a 2 GB book. The box has 200 GB/s of memory bandwidth. Show why bandwidth is not the binding constraint, then say what is.

Approach: Convert the update rate into bytes per second at cache line granularity, compare with the quoted bandwidth, then think about how many misses can be outstanding at once.

Bandwidth use is 512 MB/s against 200 GB/s, so the binding constraint is memory level parallelism, meaning the number of cache misses one core can have outstanding. Each update pulls one 64-byte cache line, so 8e6 * 64 is 5.12e8 bytes per second, a quarter of one percent of the quoted bandwidth. The real limit is that a single core has a fixed number of line fill buffers, ten to sixteen on a typical Intel core, so at most that many misses are in flight at once. With a 90 ns miss latency and 12 outstanding misses, one core sustains 12/90ns, about 133 million misses per second in the best case, and far fewer in practice because a dependent load cannot issue until the previous one returns. Random access defeats the hardware prefetcher entirely, since it detects strides and this pattern has none. The fixes are to make the access pattern predictable so prefetch works, to pack the hot fields so one line answers the whole query, to issue independent loads so several misses overlap, and to shrink the working set until it fits in L3. Quoting aggregate bandwidth to justify a random access design is the error the question is testing.

Follow-up: You add explicit software prefetch four iterations ahead. Under what condition does that make throughput worse?

Key concepts: memory bandwidth, memory level parallelism, cache line, prefetch.