Momentum and Adam are described as fixing different problems. State the specific failure each addresses, write the update rule for each, and say what Adam's bias correction is for and what happens without it.

Momentum and Adam are described as fixing different problems. State the specific failure each addresses, write the update rule for each, and say what Adam's bias correction is for and what happens without it.

Approach: Consider a quadratic with a badly conditioned Hessian for momentum, and coordinates with very different gradient magnitudes for the adaptive rate. Then expand the moving average at step one.

Momentum fixes slow progress along low-curvature directions of an ill-conditioned objective by accumulating an exponentially weighted gradient, and Adam fixes the need for a single learning rate to suit coordinates whose gradients differ by orders of magnitude by dividing each coordinate by a running root mean square. Momentum is m_t = beta*m_{t-1} + g_t with the update w <- w - eta*m_t, which sums consistent gradient components with an effective gain 1/(1 - beta), so beta = 0.9 amplifies a persistent direction tenfold while oscillating components cancel, cutting the iterations needed on a condition number kappa from order kappa to order sqrt(kappa). Adam keeps m_t = beta_1*m_{t-1} + (1 - beta_1)*g_t and v_t = beta_2*v_{t-1} + (1 - beta_2)*g_t^2 and steps by eta*m_hat/(sqrt(v_hat) + epsilon), so the per-coordinate step is scale invariant in the gradient. Both moving averages start at zero, so at step one m_1 = (1 - beta_1)*g_1 is 0.1 times the gradient and v_1 is 0.001 times the squared gradient with beta_2 near one, so both are biased toward zero. Dividing by (1 - beta^t) removes it. Without the correction, v_hat is far too small early on and the ratio produces an enormous first step, which is why an uncorrected implementation diverges in the first few hundred iterations or needs a warmup to survive.

Follow-up: Why does Adam sometimes generalise worse than tuned SGD with momentum on the same architecture, and what does the effective step size argument say about it?

Key concepts: momentum, adaptive learning rate, bias correction, condition number.