A quote engine runs at 3.0 GHz. It reads a 64-byte order book level that misses L1, L2 and L3 and misses the TLB as well. Estimate the total stall in cycles and nanoseconds, then say what changes if the same data sits in a 1 GB huge page that is already mapped.

A quote engine runs at 3.0 GHz. It reads a 64-byte order book level that misses L1, L2 and L3 and misses the TLB as well. Estimate the total stall in cycles and nanoseconds, then say what changes if the same data sits in a 1 GB huge page that is already mapped.

Approach: Cost the page walk and the DRAM access separately, add them, and convert cycles at the stated clock.

About 500 cycles, or 165 ns, and huge pages remove roughly a third of it. A four-level page walk needs up to four memory references to fetch page table entries, though the upper levels are usually cached in the paging structure caches, so the realistic TLB miss cost is one to two DRAM-ish accesses, call it 60 to 100 ns. The data access itself is a full DRAM read of one 64-byte cache line, about 80 ns on the local NUMA node. Together that is 140 to 180 ns, which at 3.0 GHz is 420 to 540 cycles of stall for one field read. A 1 GB huge page covers a billion bytes with one TLB entry rather than 262,144 four-kilobyte entries, so a working set that thrashed the TLB now sits in a handful of entries permanently and the page walk disappears, leaving the 80 ns DRAM read. It also reduces the page walk depth. The remaining fix is layout: pack the fields the hot path reads into the same cache line so one 80 ns miss serves the whole decision rather than four separate misses.

Follow-up: Your book is an array of structs and the hot path reads two of the twelve fields. What is the effective ratio of bytes fetched to bytes used, and what layout do you switch to?

Key concepts: tlb miss, cache line, huge pages, page walk.