Arm A has 400 observations with mean 0.50 and standard deviation 1.0. Arm B has 100 observations with mean 0.80 and standard deviation 3.0. Test the difference in means and explain why the pooled-variance t-test is the wrong test here and in which direction it errs.

Arm A has 400 observations with mean 0.50 and standard deviation 1.0. Arm B has 100 observations with mean 0.80 and standard deviation 3.0. Test the difference in means and explain why the pooled-variance t-test is the wrong test here and in which direction it errs.

Approach: Build the standard error from each arm's own variance, compare it with the pooled construction, and check which arm the pooling borrows precision from.

0.99. The Welch standard error of the difference is sqrt(s_A^2/n_A + s_B^2/n_B) = sqrt(1/400 + 9/100) = sqrt(0.0925) = 0.304, so t = 0.30/0.304 = 0.99 against about 105 degrees of freedom from the Welch-Satterthwaite formula, which is nowhere near significant. The pooled-variance t-test computes s_p^2 = (399*1 + 99*9)/498 = 2.59 and a standard error of sqrt(2.59*(1/400 + 1/100)) = 0.180, giving t = 1.67. Pooling is valid only under equal variances; here they differ by a factor of 9 and the high-variance arm is also the smaller one, which is exactly the configuration in which pooling lends the noisy arm the precision of the large quiet one and makes the test anti-conservative. Use Welch as the default, since it costs almost no power when the variances happen to be equal.

Follow-up: Arm B's variance is high because 3 of its 100 observations are enormous. What do you do before running any test at all?

Key concepts: Welch t-test, unequal variances, Satterthwaite degrees of freedom, standard error.