P Values, Multiple Testing and False Discoveries in Backtests

A p-value answers one narrow question, and testing two hundred signals at once turns about ten of them into false discoveries by arithmetic alone.

A p-value is the probability of seeing a test statistic this extreme, or more extreme, if the null hypothesis were actually true. It is not the probability that a strategy works, and it is not the probability that the null hypothesis is true, and confusing those three things is how a backtest full of noise ends up presented as a discovery.

The definition matters because the common misreading runs backward. A researcher who sees a p-value of 0.03 and says there is a 97 percent chance the strategy is real has inverted the statement. The correct reading is narrower: if the strategy truly had no edge, a result this strong would show up about 3 percent of the time by chance. That is a statement about how surprising the data would be under a specific assumption, not a probability attached to the strategy itself.

The distinction is not academic. A quantitative researcher who treats a low p-value as proof of an edge will keep funding strategies that are indistinguishable from noise, because noise reliably produces low p-values at a known, fixed rate. That rate is the whole subject of multiple testing.

Testing two hundred candidate signals at the five percent significance level produces about ten apparent discoveries from pure noise alone, since two hundred multiplied by 0.05 equals ten. A researcher who runs that screen, keeps the one signal that clears the bar, and writes it up without mentioning the other one hundred and ninety-nine has reported a result the null hypothesis predicted, not one that contradicts it.

The failure covered in hypothesis testing mistakes is choosing which test to report after seeing the outcome. Multiple testing correction is the formal machinery for fixing the threshold once the number of tests actually run is known, and three named methods handle it differently, at different costs.