Maximise the Gaussian log likelihood l(X) = -0.5 (log det X + tr(X^{-1} S)) over positive definite X, where S is a fixed positive definite sample scatter matrix. Give the maximiser and the matrix derivatives you used.

Maximise the Gaussian log likelihood l(X) = -0.5 (log det X + tr(X^{-1} S)) over positive definite X, where S is a fixed positive definite sample scatter matrix. Give the maximiser and the matrix derivatives you used.

Approach: Differentiate the two matrix functions separately, using the derivative of log det and the derivative of a trace against an inverse, then set the gradient to zero and check the second order behaviour along a ray.

X = S. The matrix calculus facts are d log det X = tr(X^{-1} dX), so the log determinant derivative with respect to X is X^{-1}, and d tr(X^{-1} S) = -tr(X^{-1} dX X^{-1} S) = -tr(X^{-1} S X^{-1} dX), so that gradient is -X^{-1} S X^{-1} by the trace identity. Setting the total gradient to zero gives -0.5(X^{-1} - X^{-1} S X^{-1}) = 0, and left and right multiplying by X gives X = S, the maximum likelihood estimator of a covariance. To see it is a maximum rather than a stationary point of another kind, put X = S^{1/2} Y S^{1/2} and let d_i be the eigenvalues of Y. The objective becomes a constant minus 0.5 sum(log d_i + 1/d_i), and log d + 1/d is strictly convex with its minimum at d = 1, so the objective is maximised only at Y = I, that is X = S. If S is singular, the supremum is not attained and the estimate has to be regularised.

Follow-up: Add a penalty lambda * tr(X^{-1}) to the objective. What is the maximiser now, and how does it shrink the eigenvalues of S?

Key concepts: matrix calculus, log determinant derivative, trace identity, maximum likelihood estimator.