What is Gradient Clipping?

What is Gradient Clipping?

Approach: Describe limiting the magnitude of gradients during backpropagation.

A technique to prevent the 'exploding gradient' problem. If the norm of the gradient exceeds a certain threshold, the gradient is scaled down to that threshold. This ensures updates remain stable.

Follow-up: Which type of networks often need this (e.g., RNNs)?

Key concepts: exploding gradients, norm, threshold.