L2 regularisation and weight decay coincide for plain SGD but not for Adam. Explain the mechanism that separates them and state what decoupled weight decay changes in the update rule.

L2 regularisation and weight decay coincide for plain SGD but not for Adam. Explain the mechanism that separates them and state what decoupled weight decay changes in the update rule.

Approach: Write the Adam update with an L2 term folded into the gradient, then track what the second-moment normalisation does to that term for parameters with different gradient scales.

Folding lambda*w into the gradient makes the penalty pass through Adam's second-moment normalisation, so parameters with large historical gradients have their decay divided down and are effectively penalised less, while decoupled weight decay subtracts eta*lambda*w after the adaptive step and applies the same relative shrinkage everywhere. In plain SGD the update is w <- w - eta*(g + lambda*w) = (1 - eta*lambda)*w - eta*g, so the two formulations are algebraically identical. Adam divides by sqrt(v) + epsilon where v is a running second moment of the gradient, so the coupled form gives an effective decay of eta*lambda*w/(sqrt(v) + epsilon) that depends on each parameter's own gradient history. Weights in quiet layers are then decayed hard and weights in noisy layers barely at all, which is the opposite of the intended uniform prior. Decoupling restores w <- w - eta*m_hat/(sqrt(v_hat) + epsilon) - eta*lambda*w, so the shrinkage rate becomes a clean hyperparameter tunable independently of the schedule. The symptom of getting this wrong is that the tuned lambda moves by an order of magnitude when the optimiser is swapped.

Follow-up: Under a cosine learning rate schedule the decoupled decay eta*lambda also decays. What breaks if you hold the decay constant while eta falls?

Key concepts: weight decay, adaptive learning rate, second moment, decoupled update.