You increase the minibatch size from 64 to 1024 with everything else fixed. State what happens to the gradient noise, what learning rate change keeps the training dynamics comparable, and what the…

Question

You increase the minibatch size from 64 to 1024 with everything else fixed. State what happens to the gradient noise, what learning rate change keeps the training dynamics comparable, and what the two competing scaling rules are.

Accepted Answer

The gradient variance falls as 1/B, so going from 64 to 1024 cuts it by a factor of 16 and its standard deviation by 4, and the two proposals are to raise the learning rate by 16 under the linear scaling rule or by 4 under the square root rule. Linear scaling comes from matching the total parameter displacement per epoch: with 16 times fewer steps per epoch, each step must be 16 times larger to cover the same ground, and it holds empirically over a wide range provided a warmup is used, because the very first steps at 16 times the rate are unstable. Square root scaling comes from matching the noise scale in the stochastic differential equation view, where the diffusion term is proportional to eta/sqrt(B), so holding eta/sqrt(B) fixed preserves the stationary distribution the iterates explore. The two disagree because they preserve different things, displacement versus noise, and the empirical answer is that linear scaling holds up to a critical batch size beyond which extra samples buy nothing and the curve flattens. Past that point the gradient is already accurate and the bottleneck is the step size the curvature permits. The practical consequence is that large batches lose the implicit regularisation of gradient noise, and the generalisation gap that appears is usually recovered by longer training or explicit regularisation rather than by tuning the rate alone.