In a loss landscape with d = 10^6 parameters, argue why strict local minima are rare relative to saddle points, using the eigenvalues of the Hessian. Say what that implies for which optimisation…

Question

In a loss landscape with d = 10^6 parameters, argue why strict local minima are rare relative to saddle points, using the eigenvalues of the Hessian. Say what that implies for which optimisation difficulty actually matters.

Accepted Answer

A critical point is a minimum only when all d Hessian eigenvalues are positive, and with d = 10^6 even a modest probability of any single eigenvalue being negative makes that outcome astronomically unlikely, so almost all critical points are saddles and the real obstacle is the slow crawl across near-flat regions rather than being trapped. If eigenvalue signs behaved like independent fair draws the probability of all being positive would be 2^{-d}, which is zero for practical purposes. Random matrix results sharpen this: for these landscapes the fraction of negative eigenvalues at a critical point falls as the loss value falls, so high-loss critical points are saddles with many descent directions and the few near-minima sit at low loss and have similar values to each other. That is why different random seeds reach comparably good solutions. The cost of a saddle is a plateau where the gradient is small while a descent direction still exists, so plain gradient descent slows to a crawl for many iterations. Momentum helps by accumulating the small consistent component, and gradient noise from minibatching helps by kicking the iterate along the negative curvature directions. Second-order methods that step toward a stationary point make this worse, since Newton's method is attracted to saddles, which is why saddle-free variants take the absolute value of the eigenvalues.