Hypothesis Testing Mistakes That Kill Quant Research

Screening two hundred candidate signals at the five percent level produces about ten false discoveries from noise alone, before any real edge appears.

Hypothesis testing fails in quant research mainly through the number of tests a researcher runs before reporting the one that worked. A single significant result pulled from a wide search is not evidence of an edge, it is the expected output of chance: testing two hundred candidate signals at the five percent significance level produces roughly ten apparent discoveries from noise alone, before any one of them reflects a real relationship in the market.

A p-value of five percent means that, if a signal actually has no predictive power, there is still a five percent chance its test statistic clears the significance bar by luck alone. Run two hundred independent tests on signals that are all pure noise and the expected count of false positives is two hundred multiplied by 0.05, which equals ten. A researcher who screens two hundred candidate features, momentum windows, or factor combinations and highlights the one result with a p-value under 0.05 has not beaten those odds, they have landed inside them.

The trap closes because the write-up almost never mentions the other one hundred and ninety-nine tests that failed. A memo describing one significant backtest reads as a discovery. The identical result described honestly, as one hit out of two hundred trials run at a five percent noise rate, reads as exactly what pure chance predicts, and a reviewer who asks how many configurations were tried before this one is asking the single most useful question in the whole process.

With enough observations, a coefficient of essentially zero can still clear a significance threshold. High-frequency return data supplies millions of observations, and standard errors shrink with the square root of sample size, so a predictive relationship worth a fraction of a basis point can post a p-value under 0.001 while remaining smaller than the bid-ask spread and the market impact it would cost to trade it, a gap covered in slippage and execution costs.

A desk that screens purely for low p-values will keep surfacing relationships that are statistically real and economically worthless. The fix is to set a minimum effect size before running the test at all, stated in terms that map to tradable profit after costs, so the hypothesis test answers the question that actually matters rather than confirming that a very large sample can detect a very small number.