A process holds a 20 GB order book cache and loses 15 percent throughput after moving to a two socket box. Explain what TLB pressure and NUMA placement each contribute, and give the concrete configuration changes and the counters that confirm each one.

A process holds a 20 GB order book cache and loses 15 percent throughput after moving to a two socket box. Explain what TLB pressure and NUMA placement each contribute, and give the concrete configuration changes and the counters that confirm each one.

Approach: Count how many page table entries the working set needs against the size of the translation cache, then ask which socket the memory was first touched from and where the thread runs now.

With 4 KB pages a 20 GB working set needs 5 million translations against a few thousand entries of translation cache, so almost every access takes a page walk, and 1 GB huge pages cut that to 20 entries, while poor NUMA placement adds 50 to 100 percent to any access served by the other socket memory controller. Configure explicit huge pages for the cache through a reserved hugetlbfs mount or madvise, since transparent huge pages give the same benefit with compaction stalls that appear as latency spikes. For placement, pin each worker to a core on one socket, allocate its share of the cache from that socket node, and touch the pages from the thread that will use them, because Linux places a page on the node of the first touch rather than the node of the allocation call. Confirm with the dtlb load miss counter and with numastat showing remote accesses near zero.

Follow-up: What does this analysis say about the cost of the first access to every page after a restart, and how do you remove it before the open?

Key concepts: huge pages, page walk, numa, first touch, memory placement.