You randomise a trading rule across 50 desks with 200 orders each, and the outcome has an intra-desk correlation of 0.05. Compute the design effect and the effective sample size, and say what happens to a t-test that treats the 10,000 orders as independent.

You randomise a trading rule across 50 desks with 200 orders each, and the outcome has an intra-desk correlation of 0.05. Compute the design effect and the effective sample size, and say what happens to a t-test that treats the 10,000 orders as independent.

Approach: Compute the variance inflation caused by within-cluster correlation, divide the raw count by it, and then check the limit as orders per desk grows.

10.95. With m = 200 orders per cluster and an intra-cluster correlation of rho = 0.05, the design effect is 1 + (m - 1)*rho = 1 + 199*0.05 = 10.95, so the 10,000 orders carry the information of 10000/10.95 = 913 independent observations. A t-test that treats orders as independent understates the standard error by sqrt(10.95) = 3.31, so a true t-statistic of 1.5 is reported as 5.0 and the actual size of a nominal 5% test is far above 5%. The remedy is to compute clustered standard errors at the level of randomisation, which is the desk, or equivalently to run the test on the 50 desk-level means. With only 50 clusters the cluster-robust variance is itself noisy, so use a t reference with 49 degrees of freedom or a wild cluster bootstrap. Adding orders per desk buys almost nothing beyond this point, since the effective sample size tends to 50/0.05 = 1000. Power therefore comes from randomising more desks rather than from more orders per desk.

Follow-up: You can afford 20,000 orders in total. How do you split them between number of desks and orders per desk to maximise power at rho = 0.05?

Key concepts: design effect, intra-cluster correlation, clustered standard errors, effective sample size.