You plan an A/B test of 10,000 observations per arm but check significance at the 5% level after every 1,000. Give the approximate true type I error of that procedure, explain the mechanism, and give two designs that make repeated looks legitimate.

You plan an A/B test of 10,000 observations per arm but check significance at the 5% level after every 1,000. Give the approximate true type I error of that procedure, explain the mechanism, and give two designs that make repeated looks legitimate.

Approach: Treat the running z-statistic as a random walk on an information scale and count crossings of a fixed boundary, then name the corrections that fix the boundary in advance.

0.19. Under the null the sequence of cumulative z-statistics behaves like a random walk on an information scale, and the procedure rejects if it ever crosses plus or minus 1.96, so ten chances to cross give a true type I error of roughly 19% instead of 5%. The looks are strongly dependent, which is why the figure sits far below the Bonferroni bound of 10*0.05, which is 0.5, and far above 0.05. Optional stopping also biases the reported effect upward, because you stop precisely on the looks where noise happened to help, so the published lift is larger than the truth even when the effect is real. Two legitimate designs are an alpha-spending function such as O'Brien-Fleming, which sets very strict early boundaries and spends the remaining alpha late so the total is exactly 5%, and a sequential probability ratio test with boundaries on the likelihood ratio fixed in advance. Both fix the total error rate before any data arrives.

Follow-up: Under an O'Brien-Fleming boundary with five looks, what is the nominal cutoff at the first look, and what does that cost you at the last?

Key concepts: optional stopping, type I error inflation, alpha spending, sequential testing.