Huge pages are proposed for a 40 GB in-memory tick store. Explain the two mechanisms by which they help, then give two concrete situations where transparent huge pages make latency worse and what you do instead.

Huge pages are proposed for a 40 GB in-memory tick store. Explain the two mechanisms by which they help, then give two concrete situations where transparent huge pages make latency worse and what you do instead.

Approach: Count the TLB entries needed to cover the working set at each page size, then consider what the kernel has to do at run time to produce a huge page on demand.

Huge pages help by extending TLB reach from a few megabytes to tens of gigabytes and by ending the page walk a level early, and transparent huge pages hurt when the allocation path enters direct compaction and stalls the faulting thread, or when khugepaged collapses pages in the background and takes locks the process needs. Covering 40 GB with 4 KB pages needs 10.5 million page table entries, and a data TLB holds on the order of 1,500 entries, so the working set cannot be covered and the hot path takes a page walk on most accesses. A 2 MB page covers 512 times as much per entry, so 40 GB needs 20,480 entries and a 1 GB page needs 40, which fits. The walk itself is also shorter because a huge page terminates the walk a level early. Transparent huge pages hurt in two ways. First, when memory is fragmented the allocation path may enter direct compaction, which moves pages synchronously and stalls the faulting thread for milliseconds, and this appears as a latency spike with no obvious cause. Second, khugepaged scans and collapses pages in the background, and the collapse takes locks that stall the process. The answer is to preallocate explicit huge pages at boot when memory is not yet fragmented, map the store with them at start up, set transparent huge pages to madvise or never so nothing happens implicitly, and lock the mapping so it is never reclaimed.

Follow-up: Your store is 40 GB but only 2 GB is hot and the rest is scanned once a day. Does the huge page argument still hold for the cold part?

Key concepts: huge pages, tlb reach, page walk, memory compaction.