Explain geometrically why lasso sets coefficients exactly to zero while ridge does not, using the constraint-region picture. Then give the closed form of the lasso solution under an orthonormal design and state the threshold.

Explain geometrically why lasso sets coefficients exactly to zero while ridge does not, using the constraint-region picture. Then give the closed form of the lasso solution under an orthonormal design and state the threshold.

Approach: Compare the elliptical contours of the residual sum of squares against the L1 diamond and the L2 ball, and ask where a generic contour first touches each region.

The L1 constraint region has corners on the axes so a generic elliptical contour touches it at a corner with probability bounded away from zero, giving exact zeros, and under an orthonormal design lasso is soft thresholding: b_j = sign(c_j) * max(|c_j| - lambda, 0), where c_j is the ordinary least squares coefficient. The L2 ball is smooth and rotationally symmetric, so the first contact point has all coordinates non-zero except on a measure-zero set of orientations, which is why ridge shrinks without producing sparsity while lasso selects. The same fact appears in the subgradient: the L1 penalty has subgradient [-lambda, lambda] at zero, so zero is optimal whenever the unpenalised gradient is smaller than lambda in absolute value, while the L2 penalty has derivative 2*lambda*b which is itself zero at the origin and imposes no threshold. The soft thresholding form makes the selection explicit: any OLS coefficient below lambda is set to exactly zero and the rest are pulled toward zero by lambda, so lasso biases the survivors as well as dropping the others. With correlated predictors lasso picks one of the group almost arbitrarily, which makes the selected set unstable across resamples.

Follow-up: Under an orthonormal design what is the corresponding closed form for best subset selection, and how does it differ from soft thresholding?

Key concepts: L1 constraint, corner solution, soft thresholding, sparsity.