Overfitting Detection in Machine Learning Trading Models

The deflated Sharpe ratio and the probability of backtest overfitting expose strategies tuned to noise across too many tested variations.

Overfitting detection in machine learning trading models means running a set of specific checks, the deflated Sharpe ratio, the probability of backtest overfitting, and a comparison between the number of features tested and the number of independent return observations available, before a strategy's backtest result is trusted at all. Any one of these checks failing is a reason to treat a promising backtest as noise dressed up as a discovery rather than as a real edge.

A researcher who tests 200 candidate signals against a five year backtest and reports the three that cleared a 2.0 Sharpe ratio threshold has, without necessarily intending to, run a multiple testing problem: with enough independent trials, some will clear any fixed threshold by chance alone, even if none of the 200 signals contains real predictive power. The more candidates tested, the higher the Sharpe ratio a purely random signal needs to clear before it is unusual enough to report.

This is the single largest source of published strategies that fail to reproduce out of sample. A team that does not track how many variations of a signal it tried before settling on the reported one has no way to correct for this effect, and a strategy presented without that count should be treated with real skepticism during due diligence or an interview discussion of past research.

The deflated Sharpe ratio adjusts the standard Sharpe ratio calculation for the number of trials run and for the length of the track record, producing a lower, more honest estimate of how likely the observed Sharpe ratio reflects genuine skill rather than the best outcome among many random draws. A strategy reporting an unadjusted Sharpe ratio of 2.5 after 500 backtested variations can have a deflated Sharpe ratio close to zero once the number of trials is accounted for.

Calculating it requires the researcher to honestly record how many variations were tried, which is as much a discipline and record keeping requirement as it is a statistical one. A research group that does not log every backtest run cannot apply this correction even if it wants to, which is one reason serious quantitative funds enforce backtest logging as a matter of process rather than leaving it to individual researchers.