What is the Vanishing Gradient problem?

What is the Vanishing Gradient problem?

Approach: Describe how gradients become too small during backpropagation in deep networks.

Occurs in deep neural networks when the gradients of the loss function approach zero as they are backpropagated through many layers. This prevents the weights of the early layers from changing, effectively stopping the network from learning.

Follow-up: How do ReLU or Batch Normalization help?

Key concepts: activation functions, chain rule, backpropagation.