With an orthonormal design matrix, derive the closed-form ridge and lasso solutions in terms of the OLS coefficients, then use the shape of the two constraint sets to explain why lasso produces exact zeros and ridge does not.

With an orthonormal design matrix, derive the closed-form ridge and lasso solutions in terms of the OLS coefficients, then use the shape of the two constraint sets to explain why lasso produces exact zeros and ridge does not.

Approach: With an orthonormal design the penalised objective separates coordinate by coordinate, so solve each one-dimensional problem, then read the same conclusion off the geometry of the norm balls.

Ridge scales every coefficient by 1/(1 + lambda) and lasso soft-thresholds each one at lambda/2, so lasso zeroes every coefficient whose OLS value falls below the threshold while ridge reaches zero only in the limit. With X'X = I the objective separates across coordinates. Ridge minimises (b_j - c_j)^2 + lambda*b_j^2 where c_j is the OLS coefficient, giving b_j = c_j/(1 + lambda), a proportional shrink that never hits zero at finite lambda. Lasso minimises (b_j - c_j)^2 + lambda*|b_j|, whose subgradient condition gives b_j = sign(c_j)*max(|c_j| - lambda/2, 0). In constrained form both solve least squares subject to a norm ball: the L2 ball is round with no corners, so the residual contours almost surely touch it at a point with all coordinates non-zero, while the L1 ball is a cross-polytope whose corners sit on the axes and are hit with positive probability. That geometry also explains why lasso is unstable among highly correlated regressors, since it picks one corner and the choice is nearly arbitrary.

Follow-up: Elastic net adds both penalties. What does that do to a group of five nearly identical regressors, and why?

Key concepts: ridge regression, lasso, soft thresholding, constraint set geometry.