A residual block computes y = x + F(x). Write the derivative of the loss with respect to x through one block and through L stacked blocks, and explain what that shows about training depth.

A residual block computes y = x + F(x). Write the derivative of the loss with respect to x through one block and through L stacked blocks, and explain what that shows about training depth.

Approach: Differentiate the block output with respect to its input, then multiply the per-block Jacobians and inspect the term that survives when the branch derivatives are small.

One block gives dy/dx = I + dF/dx, and L stacked blocks give a product of (I + dF_i/dx) terms whose expansion contains the pure identity path, so the gradient reaching the first block never decays to zero however small the branch derivatives become. In a plain network the gradient is a product of per-layer Jacobians, so a typical singular value below one shrinks the signal exponentially in depth. The residual form makes each factor a perturbation of the identity, and expanding the product gives I plus a sum of single-branch terms plus higher-order cross terms, so there is always a path from the loss to any block that passes through no branch weights at all. That is why depth stops being an obstacle to optimisation and networks of one hundred blocks train. The block also starts near an identity function when the branch is initialised small, so adding depth cannot make the initial function worse and the optimiser only has to learn the correction. The cost is that the forward variance grows additively across blocks unless the branch output is scaled or a normalisation layer sits inside it.

Follow-up: If each branch Jacobian has norm 0.1 and there are 100 blocks, what is the norm of the full product, and what does the binomial expansion say about which terms dominate?

Key concepts: identity path, Jacobian, gradient flow, residual block.