Derive the He initialisation variance for a ReLU layer with fan-in n. State the variance you draw the weights from, and explain what breaks if you use the Xavier variance 1/n with ReLU in a 50-layer network.

Derive the He initialisation variance for a ReLU layer with fan-in n. State the variance you draw the weights from, and explain what breaks if you use the Xavier variance 1/n with ReLU in a 50-layer network.

Approach: Track the variance of the pre-activation from one layer to the next assuming independent zero-mean weights, and account for ReLU zeroing half the distribution.

Var(w) = 2/n. He initialization draws each weight from that variance. For a pre-activation a = sum_{i=1}^{n} w_i h_i with independent zero-mean weights, Var(a) = n * Var(w) * E[h^2]. For a symmetric input distribution ReLU zeroes half the mass and keeps the rest unchanged, so E[h^2] = Var(a_prev)/2. Substituting gives Var(a) = n * Var(w) * Var(a_prev)/2, and holding the variance constant across layers requires n times Var(w) to equal 2, that is a weight variance of two over the fan-in. Using the Xavier variance 1/n instead makes the per-layer factor 1/2, so after 50 layers the forward signal variance is scaled by 2^{-50}, roughly 1e-15, and the activations are numerically zero. The backward pass is scaled by the same factor, so no gradient reaches the early layers and the network trains only its last few layers. The factor 2 is the entire correction and it exists only because ReLU discards half the distribution. Xavier's 1/n is derived for a symmetric activation with derivative near 1 at the origin, such as tanh, where the whole distribution is kept, and it targets a compromise between the forward fan-in and the backward fan-out. The same reasoning gives the gain for other activations, and residual networks add the further requirement that the branch output be scaled so the variance stays flat as blocks are added.

Follow-up: For a residual block computing x + F(x), what happens to the variance after L blocks with standard initialisation, and how is the branch scaled to prevent it?

Key concepts: He initialization, variance propagation, fan-in, ReLU.