Derive the ridge regression estimator for min ||y - Xb||^2 + lambda*||b||^2. State what happens to the fitted values as lambda goes to infinity, and what happens to each coefficient when X has orthonormal columns.

Derive the ridge regression estimator for min ||y - Xb||^2 + lambda*||b||^2. State what happens to the fitted values as lambda goes to infinity, and what happens to each coefficient when X has orthonormal columns.

Approach: Differentiate the penalised sum of squares, set the gradient to zero and solve. Then substitute X^T X = I to see the per-coefficient effect.

b_ridge = (X^T X + lambda*I)^{-1} X^T y, the fit collapses to the mean as lambda goes to infinity, and under an orthonormal design each coefficient is the OLS one divided by (1 + lambda). Differentiating gives the normal equations -2 X^T (y - Xb) + 2*lambda*b = 0, so (X^T X + lambda*I) b = X^T y. Adding lambda*I raises every eigenvalue of X^T X by lambda, so the inverse exists even when X is rank deficient, which is why ridge is defined with more features than observations. As lambda grows the inverse behaves like I/lambda and b goes to zero, so only the intercept survives. With X^T X = I the solution is b_j = c_j/(1 + lambda), where c_j is the ordinary least squares coefficient, a uniform shrinkage factor that never reaches exactly zero for finite lambda. In the singular value basis the shrinkage is d_i^2/(d_i^2 + lambda), so directions with small singular values, the ones the sample estimates worst, are shrunk hardest.

Follow-up: What is the effective degrees of freedom of a ridge fit, and how would you use it to compare two values of lambda on the same design?

Key concepts: ridge regression, normal equations, shrinkage factor, orthonormal design.