Give three quantitative diagnostics that distinguish a genuinely predictive backtest from an overfitted one, using only the backtest output and the research process log. For each, state the number you would compute and what value would concern you.

Give three quantitative diagnostics that distinguish a genuinely predictive backtest from an overfitted one, using only the backtest output and the research process log. For each, state the number you would compute and what value would concern you.

Approach: Look for signatures that only overfitting produces: performance collapsing out of sample, sensitivity to parameters, and concentration of the return in a few observations.

Compute the ratio of out-of-sample Sharpe to in-sample Sharpe, the parameter surface around the chosen configuration, and the concentration of the profit and loss across time and trades, and treat a ratio below about 0.5, a sharp isolated peak, or a top-5% of days carrying most of the return as evidence of overfitting. The out-of-sample degradation ratio is the most direct measure: under a real edge the out-of-sample Sharpe should be lower only by the estimation noise, so a fall from 2.4 to 0.3 is a selection artefact. The parameter surface matters because a real effect is a plateau: neighbouring lookbacks and thresholds should perform similarly, and a configuration that works only at exactly 17 days with 12 and 22 both failing is fitted to the noise. Return concentration exposes a third failure: compute the share of total profit contributed by the best 1% of days and the maximum single-trade contribution, and if removing five days flips the strategy negative then the Sharpe estimate rests on five observations and the standard error is far larger than the formula assumes. Alongside these, record the trial count, meaning every configuration ever run including the abandoned ones, and deflate the reported statistic by the expected maximum under the null. A backtest with no record of how many variants preceded it cannot be evaluated at all.

Follow-up: How would you construct a synthetic null by resampling the strategy's own trade sequence, and what does that add over the parametric Sharpe standard error?

Key concepts: out-of-sample degradation, parameter sensitivity, return concentration, trial count.