Write out the bias-variance decomposition of the expected squared error at a fixed test point x for a model trained on random samples with noise variance sigma^2. State each of the three terms and prove the cross terms vanish.

Write out the bias-variance decomposition of the expected squared error at a fixed test point x for a model trained on random samples with noise variance sigma^2. State each of the three terms and prove the cross terms vanish.

Approach: Insert and subtract the average prediction E[f_hat(x)] inside the squared error, expand, and take expectations over both the training sample and the label noise.

E[(y - f_hat(x))^2] = sigma^2 + (E[f_hat(x)] - f(x))^2 + Var(f_hat(x)), that is irreducible noise plus bias squared plus variance. Write y = f(x) + eps with E[eps] = 0 and Var(eps) = sigma^2, and let f_bar = E[f_hat(x)] over training samples. Then y - f_hat = eps + (f - f_bar) + (f_bar - f_hat). Squaring gives three squares and three cross terms. Every cross term containing eps vanishes because the noise at the test point is independent of the training sample and has mean zero. The remaining cross term 2(f - f_bar)(f_bar - f_hat) has expectation zero because f - f_bar is a constant and E[f_bar - f_hat] vanishes by the definition of f_bar. What is left is sigma^2 + bias^2 + variance. The decomposition is specific to squared loss, and sigma^2 is a floor no model can beat, which is why a reported R-squared near one on daily returns is normally a sign of leakage rather than skill.

Follow-up: For k-nearest neighbours averaging k neighbours with noise variance sigma^2, what is the variance term explicitly, and how does the bias behave as k grows?

Key concepts: bias squared, variance, irreducible noise, cross term.