Dropout with keep probability p is described as training an ensemble. State how many networks that ensemble contains for n hidden units, why test-time scaling by p approximates the ensemble prediction, and where the approximation is exact.

Dropout with keep probability p is described as training an ensemble. State how many networks that ensemble contains for n hidden units, why test-time scaling by p approximates the ensemble prediction, and where the approximation is exact.

Approach: Count the sub-networks generated by independent unit masks, then compare the average of the sub-network outputs with the single scaled network for a linear and then a nonlinear unit.

There are 2^n sub-networks, and multiplying activations by p at test time reproduces the ensemble average exactly for a linear model and the normalised geometric mean of the sub-network outputs exactly for a single-layer softmax. Each of the n units is independently kept or dropped, so a mask is a binary vector and the ensemble has 2^n members sharing one weight set. Averaging 2^n forward passes is not feasible, so the weight scaling rule replaces it: the expected input to a downstream unit under dropout is p times the full activation, so scaling by p matches the first moment. For a linear output the prediction is linear in the activations, the expectation passes straight through, and the scaled network output equals the ensemble mean. With deep nonlinearities neither identity holds and the rule is a first-order approximation that works because the variance of the masked sum is small relative to its mean. Dropout also penalises co-adapted features, which is why the surviving weights grow as p falls, and why dropout and weight decay have to be tuned together rather than separately.

Follow-up: For a linear regression with dropout on the inputs, what deterministic penalty does the expected loss equal, and how does it differ from plain ridge?

Key concepts: dropout, weight scaling rule, geometric mean, ensemble average.