Adding a column of pure noise to a regression with n = 100 observations and 5 regressors raises R-squared from 0.400 to 0.403. Prove that R-squared can never fall when a regressor is added, compute adjusted R-squared before and after, and say which model you keep.

Adding a column of pure noise to a regression with n = 100 observations and 5 regressors raises R-squared from 0.400 to 0.403. Prove that R-squared can never fall when a regressor is added, compute adjusted R-squared before and after, and say which model you keep.

Approach: Argue from the nesting of the two column spaces that the minimised residual sum of squares cannot rise, then apply the degrees of freedom penalty.

R-squared cannot fall because the smaller model is nested inside the larger one, and adjusted R-squared falls from 0.368 to 0.364, so keep the five-regressor model. OLS minimises the residual sum of squares over the column space of X. Adding a column enlarges that space, and setting the new coefficient to zero reproduces the old fit exactly, so the minimised residual sum of squares weakly decreases and R-squared = 1 - RSS/TSS weakly increases whatever the new column contains. Adjusted R-squared is 1 - (1 - R^2)*(n - 1)/(n - k - 1): at k = 5 it is 1 - 0.600*99/94 = 0.368, and at k = 6 it is 1 - 0.597*99/93 = 0.364. The penalty for the extra degrees of freedom outweighs 0.003 of fit, which is the same statement as the new regressor having |t| < 1. Adjusted R-squared is a weak filter, so confirm with an out-of-sample test or an information criterion.

Follow-up: With 100 observations and 100 regressors of pure noise, what is R-squared, and what does that say about the in-sample fit of any high-dimensional model?

Key concepts: R-squared, adjusted R-squared, residual sum of squares, degrees of freedom.