You run 200 independent backtests of strategies that have no true edge. Give the probability that at least one produces a p-value below 0.001, then state what Bonferroni and Benjamini-Hochberg each do to your keep rule and what quantity each one controls.

You run 200 independent backtests of strategies that have no true edge. Give the probability that at least one produces a p-value below 0.001, then state what Bonferroni and Benjamini-Hochberg each do to your keep rule and what quantity each one controls.

Approach: Treat the 200 p-values as independent uniforms under the null and compute a complement, then state the two correction rules and the error quantity each targets.

0.1814. Under the null each p-value is uniform, so each test clears 0.001 with probability 0.001 and none does with probability 0.999^200 = 0.8186, leaving 0.1814. At the ordinary 5% cutoff you would expect 200*0.05 = 10 false keeps. Bonferroni holds the family-wise error rate, the chance of any false keep at all, at 5% by testing each hypothesis at 0.05/200 = 0.00025. That threshold is severe and costs power. Benjamini-Hochberg controls the false discovery rate, the expected share of false keeps among those kept: sort the p-values ascending and keep everything up to the largest k whose p-value stays under the boundary 0.05*k/200. It lets more true signals through at the price of a bounded proportion of false ones, which is the quantity a research pipeline that will retest survivors actually cares about.

Follow-up: How does the Benjamini-Hochberg procedure have to be modified when the 200 test statistics are positively correlated because the strategies share factor exposures?

Key concepts: family-wise error rate, false discovery rate, Bonferroni correction, Benjamini-Hochberg.