Explain the vanishing and exploding gradient problem in a deep network through the product of Jacobians. Give the condition on the singular values that separates the two cases, and say why sigmoid activations make vanishing gradients worse than ReLU.

Explain the vanishing and exploding gradient problem in a deep network through the product of Jacobians. Give the condition on the singular values that separates the two cases, and say why sigmoid activations make vanishing gradients worse than ReLU.

Approach: Write the gradient at layer one as a product of per-layer Jacobians, then bound its norm by the product of the largest singular values and see how it behaves in the depth.

The gradient reaching layer one is a product of L per-layer Jacobians, so its magnitude scales like the product of their singular values and therefore decays or grows exponentially in depth: a typical singular value below 1 vanishes, above 1 explodes, and only a value near 1 keeps the signal usable. Each layer contributes a Jacobian W^T * diag(phi'(a)), so the norm of the full product is bounded by the product of the per-layer norms, and if that per-layer norm averages 0.9 then over 50 layers the factor is 0.9^50, which is about 0.005, and over 100 layers 3e-5. The sigmoid derivative is p(1 - p), which has maximum 0.25 at the origin and falls toward zero as the unit saturates, so even with well-scaled weights the activation factor alone contributes at most 0.25 per layer and the product collapses within a handful of layers. ReLU has derivative exactly 1 on the active side and 0 on the other, so it neither shrinks nor amplifies the passing gradient on the active path, and the surviving problem is dead units rather than uniform decay. Exploding gradients appear when the weight singular values exceed 1, and are handled by clipping the global gradient norm, which changes the direction only when the threshold binds. Structural fixes are initialisation targeting a per-layer gain of 1, normalisation layers, and residual connections whose identity path has Jacobian I.

Follow-up: For a recurrent network with a shared weight matrix, what condition on its spectral radius makes the gradient stable, and why does that condition also limit what the network can remember?

Key concepts: Jacobian product, singular value, exponential decay, activation derivative.