Derive principal component analysis as an eigenproblem: show that the direction maximising the variance of a projection of centred data is the top eigenvector of the covariance matrix, and state what the eigenvalue equals.

Derive principal component analysis as an eigenproblem: show that the direction maximising the variance of a projection of centred data is the top eigenvector of the covariance matrix, and state what the eigenvalue equals.

Approach: Maximise the variance of w^T x subject to ||w|| = 1 with a Lagrange multiplier, and read the stationarity condition.

The maximising direction is the top eigenvector of the covariance matrix S, and its eigenvalue is exactly the variance of the data projected onto it. For centred data the variance of the projection w^T x is w^T S w, and maximising it without a norm constraint is unbounded, so impose ||w||^2 = 1 and form the Lagrangian w^T S w - lambda(w^T w - 1). Setting the derivative to zero gives 2Sw - 2*lambda*w = 0, that is Sw = lambda*w, so any stationary point is an eigenvector. Substituting back gives w^T S w = lambda, so the objective value equals the eigenvalue and the maximum is attained at the largest one. Later components repeat the argument with the added constraint of orthogonality to the previous directions, which selects the eigenvectors in descending eigenvalue order, and the total variance is the trace of S, so the share explained by component k is lambda_k divided by that trace. Because S depends on the units of each column, PCA on unstandardised features is dominated by whichever column has the largest scale, and on a panel of returns the components are only meaningful once the series are on comparable scales.

Follow-up: How do the eigenvectors of the covariance matrix relate to the singular vectors of the centred data matrix, and which computation is preferred numerically?

Key concepts: covariance matrix, eigenvector, Lagrange multiplier, explained variance.