Testing Strategies for Quantitative Research Code

The strongest tests in research code check a calculation against a known analytical answer on synthetic data.

The highest value tests in quantitative research code assert a known analytical answer against a calculation run on synthetic data, rather than only checking that a function runs without raising an error, because a bug in research code almost never produces a crash. It produces a plausible-looking number that is simply wrong, and a test suite built only around checking that code executes will miss exactly the class of bug that matters most in this domain.

A software bug in a typical application often manifests as a crash, an exception, or an obviously wrong output such as a blank page or a null value, which makes it visible quickly. A bug in a quantitative calculation, by contrast, frequently produces a number that is wrong by a moderate amount, a sign flip in one term, a variance computed with the wrong denominator, a shift in an index that is off by one, and every one of these still returns a float that looks like a plausible Sharpe ratio or a plausible volatility estimate.

This is the reason testing research code needs a different default than testing typical application code: the test needs to check the actual numerical value against a known, independently derived answer, not merely check that the function returned something of the right type without an error.

The strongest test for a calculation is one built on synthetic data where the correct answer can be derived by hand or from a known closed-form result, independent of the code being tested. A volatility calculation, for example, can be tested against a synthetic price series constructed from a known, fixed daily return, where the annualized volatility that the function should return can be computed directly from the definition rather than from the function itself, so the test does not simply check that the function agrees with itself.

The same approach applies to a rolling calculation: generate a synthetic series with a known, constant underlying value plus no noise, run the rolling calculation, and check that it returns exactly the underlying value once the window has filled, which catches an off-by-one error in the window boundary or an incorrect denominator far more reliably than running the function on real market data and eyeballing whether the output looks reasonable.