Of the strategy ideas your team tests, 10% have a real edge. Your test has 5% size and 60% power. An idea comes back significant at the 5% level. Give the probability it has no real edge, and state what the p-value did and did not measure.

Of the strategy ideas your team tests, 10% have a real edge. Your test has 5% size and 60% power. An idea comes back significant at the 5% level. Give the probability it has no real edge, and state what the p-value did and did not measure.

Approach: Count outcomes in a hypothetical population of ideas, splitting them by whether the edge is real and whether the test fires, then take the ratio Bayes' rule asks for.

3/7. Of 1000 ideas, 100 have an edge and 60 of those clear the test at 60% power, while 900 are dead and 45 of those clear it at 5% size. So 105 come back significant and 45 of them are false, giving 45/105 = 3/7 = 0.4286. The p-value is P(data at least this extreme | no edge); it is not P(no edge | data), and converting one into the other requires the base rate of 10% and the power of 60%, neither of which appears in the p-value. Raising power to 90% would give 90 true against the same 45 false and cut the false share to 1/3, so power does as much work here as the significance threshold. If the team's hit rate falls to 1%, the same test yields 6 true discoveries against 49.5 false ones, so most significant results are wrong while the test itself is unchanged.

Follow-up: Your team decides to raise the bar to p < 0.005 without changing anything else. What happens to the false discovery share, and what does it cost?

Key concepts: base rate, false discovery rate, statistical power, p-value.