You have five plausible models with in-sample marginal likelihoods and are asked to combine them rather than pick one. State what Bayesian model averaging computes, what the weights are, and give the specific circumstance where averaging beats selecting.

You have five plausible models with in-sample marginal likelihoods and are asked to combine them rather than pick one. State what Bayesian model averaging computes, what the weights are, and give the specific circumstance where averaging beats selecting.

Approach: Write the predictive distribution as a sum over models weighted by their posterior probabilities, then consider what the weights do as the sample grows.

Bayesian model averaging computes p(y_new | data) = sum_m p(y_new | M_m, data) * p(M_m | data), with weights proportional to the marginal likelihood of each model times its prior, and it beats selection when genuine model uncertainty remains, which is when several models have comparable marginal likelihoods. The marginal likelihood integrates the likelihood over the parameter prior, so it automatically penalises a model whose extra flexibility is not repaid by fit, which is the built-in complexity penalty that makes the weights meaningful. Picking the single best model throws away the fact that the second model was almost as probable, and the resulting predictive intervals ignore the uncertainty about which model is right, so they are too narrow. Averaging keeps that uncertainty and typically improves the predictive log score. The circumstance where it helps is exactly where the weights are diffuse, so on a short noisy return sample where five factor specifications fit almost equally well. As the sample grows the weights concentrate on one model and averaging converges to selection, so the benefit disappears at large n. The weights assume the true model is among those considered, and when it is absent they concentrate on the closest member, so stacking on out-of-sample performance is the safer alternative.

Follow-up: How is the marginal likelihood sensitive to a diffuse parameter prior, and what does that imply about comparing models with different prior widths?

Key concepts: posterior model probability, marginal likelihood, predictive distribution, model uncertainty.