Memory Allocation Strategies in Latency Sensitive Code

General purpose allocators can lock, fault a page or fragment a long-running heap, so hot paths pre-allocate everything and never call new at runtime.

Memory allocation strategies in latency sensitive code exist because calling new or malloc on a hot path introduces latency that cannot be bounded in advance. A general purpose allocator can internally take a lock shared with other threads, touch a page of memory that is not currently resident and trigger a page fault, or search a fragmented heap for a block of the right size, and any of those outcomes can turn what looks like a single instruction in the source code into a delay measured in microseconds, which is unacceptable on a path where the entire latency budget is a few hundred nanoseconds.

The standard replacement is to allocate every object a system will ever need at startup, before the market opens and before any latency-sensitive processing begins, and to hold that memory in a pool the hot path draws from and returns to at runtime. An order object pool, for instance, is sized to the maximum number of outstanding orders the strategy could plausibly have live at once, and acquiring an object from the pool becomes a matter of popping an index off a pre-sized free list rather than asking the operating system for fresh memory.

Because the pool never grows or shrinks while trading is live, its performance is fully predictable, and profiling a system built this way shows almost no variance in allocation cost across the trading day, in contrast to a general purpose allocator whose behavior changes as the heap fragments over hours of continuous operation.

An arena allocator reserves one large contiguous block of memory up front and hands out slices of it sequentially, resetting the entire arena at once, typically once per message batch or once per trading session, rather than freeing individual objects one at a time. A slab allocator is a related technique specialized for objects of a single fixed size, maintaining a free list of equally sized slots so that allocation and deallocation both become simple pointer arithmetic with none of the size-class bookkeeping a general allocator performs.

Both techniques trade flexibility for predictability. A system built around arenas and slabs cannot gracefully handle an unexpected surge in allocation beyond what was provisioned, so capacity planning has to be conservative, and that trade-off is itself something interviewers ask candidates to reason through explicitly rather than assume away.