Derive the E step and the M step of expectation-maximisation for a Gaussian mixture with K components. State the responsibility formula, the updated mean and mixing weight, and explain why the likelihood cannot decrease.

Derive the E step and the M step of expectation-maximisation for a Gaussian mixture with K components. State the responsibility formula, the updated mean and mixing weight, and explain why the likelihood cannot decrease.

Approach: Introduce the latent component indicator, take the posterior over it given current parameters, then maximise the expected complete-data log-likelihood and compare with a lower bound argument.

The E step computes the responsibility r_ik = pi_k N(x_i | mu_k, Sigma_k) / sum_j pi_j N(x_i | mu_j, Sigma_j), and the M step sets N_k = sum_i r_ik, mu_k = (1/N_k) sum_i r_ik x_i, Sigma_k = (1/N_k) sum_i r_ik (x_i - mu_k)(x_i - mu_k)^T and pi_k = N_k/n, and the observed log-likelihood rises or stays flat because each step either tightens or maximises a lower bound on it. Writing the log-likelihood with the latent indicator z_i and any distribution q over it, Jensen's inequality gives log p(x) >= E_q[log p(x, z)] + H(q), with equality exactly when q is the posterior p(z | x). The E step therefore makes the bound tight at the current parameters, and the M step maximises the bound in the parameters, so the log-likelihood after the M step is at least the bound value, which equals the previous log-likelihood. The updates are the sample statistics with each point weighted by its responsibility, which is why they look like the closed-form Gaussian fit. Monotone improvement does not mean convergence to the global maximum: EM stops at a local optimum determined by the initialisation, and the likelihood is unbounded because driving one component onto a single point with a shrinking variance sends it to infinity, so a variance floor or a prior is required in practice.

Follow-up: For a mixture fitted to daily returns, how would you choose K, and what does the unbounded likelihood imply about using the maximised likelihood to compare K?

Key concepts: responsibility, expected complete-data log-likelihood, lower bound, mixing weight.