A random forest and a gradient boosting model are built from trees on the same data. Say which primarily reduces bias and which reduces variance, what each does to the other component, and which one…

Question

A random forest and a gradient boosting model are built from trees on the same data. Say which primarily reduces bias and which reduces variance, what each does to the other component, and which one overfits as the number of trees grows.

Accepted Answer

Random forests reduce variance by averaging deep low-bias high-variance trees, gradient boosting reduces bias by sequentially fitting shallow high-bias trees to residuals, and only boosting overfits as the number of trees grows. In a forest every tree targets the same label on a bootstrap sample, so the ensemble mean carries the same bias as one tree while its variance falls toward rho*sigma^2. Adding trees can only lower that variance, so the tree count is not really a tuning parameter. In boosting each tree targets what the ensemble has not yet explained, so the training loss falls monotonically and the ensemble bias shrinks toward zero while the variance grows, because the fit increasingly encodes the sample's noise. That makes the number of rounds the single most important hyperparameter and one that must be chosen out of sample. Forests want deep trees and boosting wants shallow ones for the same reason: forests need low bias per member and can absorb the variance, boosting needs low variance per member and removes bias by iteration. On noisy financial targets boosting typically needs a small learning rate, early stopping on a purged validation split, and a depth of three or less.