You will run a two-sided A/B test at the 5% level with 80% power, 5,000 observations per arm, and a per-observation standard deviation of 2.0. Compute the minimum detectable difference in means, and say how the sample requirement scales if you want to halve it.

You will run a two-sided A/B test at the 5% level with 80% power, 5,000 observations per arm, and a per-observation standard deviation of 2.0. Compute the minimum detectable difference in means, and say how the sample requirement scales if you want to halve it.

Approach: Build the standard error of the difference in means, then place the alternative far enough beyond the critical value to leave the stated power in the upper tail.

0.112. The standard error of the difference in means is sqrt(2*sigma^2/n) = sqrt(2*4/5000) = sqrt(0.0016) = 0.04. A two-sided test at alpha = 0.05 rejects beyond 1.96 standard errors, and 80% power requires the true difference to sit a further 0.84 standard errors past that point, so the minimum detectable effect is (1.96 + 0.84)*0.04 = 0.112. In units of the outcome standard deviation that effect size is 0.112/2.0 = 0.056. That is a small effect. The required n scales with the inverse square of the effect, so halving the detectable difference to 0.056 needs four times the observations, 20,000 per arm.

Follow-up: The metric is a per-order profit with a long right tail and kurtosis near 30. What does that do to the power calculation you just did?

Key concepts: minimum detectable effect, statistical power, standard error, effect size.