State the evidence lower bound used in variational inference, show that maximising it is equivalent to minimising a KL divergence to the true posterior, and explain what the resulting approximation…

Question

State the evidence lower bound used in variational inference, show that maximising it is equivalent to minimising a KL divergence to the true posterior, and explain what the resulting approximation systematically gets wrong.

Accepted Answer

The bound is ELBO(q) = E_q[log p(x, z)] - E_q[log q(z)] and log p(x) = ELBO(q) + KL(q || p(z | x)), so maximising the bound minimises the reverse KL divergence to the posterior, and the systematic failure is that the fitted q is too narrow and misses posterior mass. The identity follows by writing log p(x) = log integral q(z) p(x, z)/q(z) dz and splitting the expectation of the log ratio into the bound plus the divergence, and because the divergence is non-negative the bound is a true lower bound, tight exactly when q equals the posterior. The direction of the divergence is the source of the bias: KL(q || p) integrates against q, so it is heavily penalised wherever q puts mass in a region the posterior does not, and barely penalised where the posterior has mass that q ignores. The optimiser therefore prefers to cover one mode tightly, and on a multimodal posterior it selects a single mode and reports credible intervals that are too narrow. For a trading model that matters directly, since parameter uncertainty is understated and any sizing that depends on it is too aggressive. A mean-field q that factorises across parameters compounds the effect by discarding posterior correlations, which shrinks the implied variance of any linear combination of parameters. Markov chain sampling has the opposite trade, being unbiased in the limit and far more expensive per unit of accuracy.