What is 'Stochastic Gradient Descent' (SGD) with Momentum?

What is 'Stochastic Gradient Descent' (SGD) with Momentum?

Approach: Describe adding a fraction of the previous update to the current one.

A technique that helps accelerate SGD in the relevant direction and dampens oscillations by adding a 'momentum' term (moving average of gradients) to the update rule.

Follow-up: What is Nesterov Accelerated Gradient (NAG)?

Key concepts: velocity, dampening oscillations, convergence.