A dual-socket box has 2 sockets of 32 cores. Your feed handler runs on socket 0, the NIC is on socket 0, and the order book lives in memory allocated by a thread that ran on socket 1. Quantify the penalty and describe the three separate mistakes this configuration makes.

A dual-socket box has 2 sockets of 32 cores. Your feed handler runs on socket 0, the NIC is on socket 0, and the order book lives in memory allocated by a thread that ran on socket 1. Quantify the penalty and describe the three separate mistakes this configuration makes.

Approach: Trace where the memory physically lives against where it is read from, and remember the first-touch allocation policy.

Every book read crosses the interconnect and costs roughly 140 ns instead of 80 ns, so hot path memory access is close to twice as slow. The first mistake is the allocation itself: Linux allocates a page on the node of the thread that first touches it, not the thread that called malloc, so a setup routine running on socket 1 places the entire book on the far node and it stays there for the life of the process. The second mistake is the split between NIC and consumer: if the NIC's DMA target buffers were also allocated on the wrong node, every arriving packet crosses the interconnect once on the way in and again when read. The third is bandwidth. Local memory bandwidth is per socket, so pushing the feed handler's traffic across the link consumes a shared resource that the other socket's work is also using, and under load the interconnect, not the DRAM, becomes the limit. The fix is to pin the handler thread to a core on socket 0, allocate and first-touch the book from that thread, bind the NIC's queues and interrupt affinity to socket 0, and verify with numactl and the per-node miss counters rather than assuming.

Follow-up: You must run two independent strategies on the two sockets sharing one NIC. How do you lay out queues and memory so neither pays a remote access on its hot path?

Key concepts: numa, first touch, interconnect, memory bandwidth.