What is 'Orthogonal Initialization'?

What is 'Orthogonal Initialization'?

Approach: Describe initializing neural network weights as orthogonal matrices.

A weight initialization technique where weights are initialized as random orthogonal matrices. It helps maintain the norm of the gradient during backpropagation, mitigating vanishing/exploding gradients.

Follow-up: In which type of networks is this particularly useful? (RNNs)

Key concepts: eigenvalues, gradient flow, vanishing gradient.