Prove that the ordinary least squares estimator is the maximum likelihood estimator when the errors are independent Gaussian with constant variance. Then state the MLE of sigma^2 and why it is biased.

Prove that the ordinary least squares estimator is the maximum likelihood estimator when the errors are independent Gaussian with constant variance. Then state the MLE of sigma^2 and why it is biased.

Approach: Write the Gaussian likelihood of the residuals, take logs, and identify which terms depend on the coefficient vector.

Maximising the Gaussian likelihood is equivalent to minimising the residual sum of squares, so the maximum likelihood estimator equals the least squares estimator, and the MLE of sigma^2 is RSS/n, which is biased low. With y_i = x_i^T b + eps_i and eps_i ~ N(0, sigma^2) independent, the log-likelihood is -n/2 * log(2*pi*sigma^2) - (1/(2*sigma^2)) * sum_i (y_i - x_i^T b)^2. Only the last term involves b, and it enters with a negative sign and a positive factor, so maximising over b minimises the residual sum of squares and gives b_hat = (X^T X)^{-1} X^T y. Differentiating in sigma^2 at that b_hat gives sigma_hat^2 = RSS/n. It is biased because the residuals are measured against fitted coefficients rather than true ones, so they are systematically smaller: E[RSS] = (n - p) sigma^2, and the unbiased estimator divides by n - p. Under Laplace errors the same argument produces least absolute deviations, which is the honest reason to use an L1 loss on fat-tailed returns.

Follow-up: If the noise variance is proportional to a known weight w_i, what does the maximum likelihood argument give, and how does it change the estimator?

Key concepts: maximum likelihood, log-likelihood, least squares, biased variance estimate.