You bag B = 100 trees whose individual predictions have variance sigma^2 and pairwise correlation rho = 0.4. Compute the variance of the bagged average as a multiple of sigma^2, and state the limiting value as B goes to infinity.

You bag B = 100 trees whose individual predictions have variance sigma^2 and pairwise correlation rho = 0.4. Compute the variance of the bagged average as a multiple of sigma^2, and state the limiting value as B goes to infinity.

Approach: Use the variance of an average of equicorrelated variables, then take the limit in the number of learners.

0.406*sigma^2. The variance of an average of B equicorrelated variables is rho*sigma^2 + (1 - rho)*sigma^2/B, so with rho = 0.4 and B = 100 it is 0.4*sigma^2 + 0.6*sigma^2/100 = 0.406*sigma^2. As B goes to infinity the second term vanishes and the variance floors at rho*sigma^2 = 0.4*sigma^2, so adding trees past a few hundred buys almost nothing. This is the whole argument for random forests: bagging alone leaves the trees highly correlated because they see nearly the same data and choose nearly the same splits, so the floor dominates. Sampling a random subset of features at each split lowers rho and therefore lowers the floor, at the cost of raising each tree's own variance and bias slightly. The trade is worth making whenever the reduction in rho beats the increase in sigma^2. On financial panels the trees are correlated because returns across assets share common factors, so rho is high and the achievable floor is worse than the same calculation on independent rows suggests.

Follow-up: If lowering the feature subsample raises each tree's variance by 20% while cutting rho from 0.4 to 0.25, does the ensemble variance improve?

Key concepts: variance of an average, correlation, bagging, limiting variance.