Explain gradient boosting as gradient descent in function space. State what the trees are fitted to at stage m, what the shrinkage parameter corresponds to, and what changes when the loss is absolute error rather than squared error.

Explain gradient boosting as gradient descent in function space. State what the trees are fitted to at stage m, what the shrinkage parameter corresponds to, and what changes when the loss is absolute error rather than squared error.

Approach: Treat the model as a point in function space, take the negative functional derivative of the empirical loss at the current fit, and identify what a tree fitted to it approximates.

At stage m the tree is fitted to the negative gradient of the loss with respect to the current predictions, the pseudo-residuals r_i = -dL(y_i, F(x_i))/dF(x_i), and the shrinkage parameter is the step size of that descent. The model F_m = F_{m-1} + nu * h_m is a step in the space of functions evaluated at the training points, and the tree h_m is the closest representable direction to steepest descent, found by least squares against r. For squared loss the pseudo-residual is exactly y_i - F(x_i), which is why the textbook case is described as fitting the residuals. For absolute error the derivative is sign(y_i - F(x_i)), so the tree is fitted to signs and the leaf values are then set to the median of the residuals in each leaf by a per-leaf line search, which is what makes the method robust to outliers. Huber loss switches between the two at a quantile-based cut. The rate nu trades steps against step size, and halving nu while doubling the number of trees usually generalises better because each move along a noisy gradient estimate is smaller. Second-order implementations use the gradient and the Hessian per row and solve the leaf value in closed form.

Follow-up: For log-loss, what is the pseudo-residual, and what closed-form leaf value does a second-order boosting implementation with L2 leaf regularisation use?

Key concepts: functional gradient, pseudo-residuals, learning rate, line search.