NumPy and Pandas Patterns That Speed Up Research

Vectorised operations, narrow dtypes and categorical columns are the patterns that keep a research pipeline fast at scale.

The NumPy and pandas patterns that make the largest difference to a research pipeline are ordinary: use vectorised operations across the whole array, avoid apply on an axis that already has a native alternative, and let pandas broadcast a computation across a column-oriented frame rather than stepping through it row by row. None of these patterns are exotic, but a strategy backtest processing a year of United States equity tick data, which runs into the billions of rows once quotes and trades are combined, magnifies the cost of skipping them from an unnoticed delay in a small script into an hours-long research run that blocks everyone waiting on the result.

The single highest value change in most research code is replacing a Python for loop, or a pandas apply call wrapping a Python function, with an operation expressed directly on the underlying array. Computing a spread from a bid and an ask column, for example, should be a single subtraction between two Series rather than a function applied row by row, because the subtraction runs inside NumPy's compiled code while apply calls back into the Python interpreter for every single row.

The difference is not marginal at the size a research pipeline actually operates at. A vectorised subtraction across ten million rows completes in a fraction of a second, while the equivalent apply call can take tens of seconds, and that gap compounds across every step of a pipeline with a dozen similar transformations chained together.

Pandas ships native, vectorised implementations of most common transformations: string operations under the str accessor, date arithmetic under the dt accessor, and rolling statistics under the rolling and expanding accessors. Reaching for apply with a lambda is usually a sign that a native method exists and was not found, and it is worth checking the accessor methods before writing a custom function.

There is a genuine exception: when the transformation is not expressible as a composition of existing vectorised operations, for instance a stateful calculation that depends on a running value carried from one row to the next. In that specific case, a compiled path through Numba or Cython usually beats both apply and a hand-written vectorised approximation, and it is worth reaching for one of those tools rather than forcing a stateful problem into a shape it does not fit.