mlsanityData to 7 Oct 2026

On the Generalization Benefit of Noise in Stochastic Gradient Descent

arXiv 2006.15081 ↗First posted 26 Jun 2020cs.LG stat.MLcited by 81

Not on the map. Why?

Abstract (arXiv)

It has long been argued that minibatch stochastic gradient descent can generalize better than large batch gradient descent in deep neural networks. However recent papers have questioned this claim, arguing that this effect is simply a consequence of suboptimal hyperparameter tuning or insufficient compute budgets when the batch size is large. In this paper, we perform carefully designed experiments and rigorous hyperparameter sweeps on a range of popular models, which verify that small or moderately large batch sizes can substantially outperform very large batches on the test set. This occurs even when both models are trained for the same number of iterations and large batches achieve smaller training losses. Our results confirm that the noise in stochastic gradients can enhance generalization. We study how the optimal learning rate schedule changes as the epoch budget grows, and we provide a theoretical account of our observations based on the stochastic differential equation perspective of SGD dynamics.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 18; 2023: 16; 2024: 14; 2025: 9; 2026: 11.

81 papers.

Sort:

By area:

  1. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  2. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  3. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  4. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  5. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  6. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  7. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  8. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  9. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  10. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  11. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  12. 2021Survey: Leakage and Privacy at Inference Timecited by 44
38 more
  1. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  2. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  3. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  4. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  5. 2025Scaling Behavior of Discrete Diffusion Language Modelscited by 32
  6. 2025Pre-training under infinite computecited by 24
  7. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  8. 2025Generalization through variance: how noise shapes inductive biases in diffusion modelscited by 23
  9. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  10. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  11. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  12. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  13. 2022VICE: Variational Interpretable Concept Embeddingscited by 15
  14. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  15. 2021Citadel: Protecting Data Privacy and Model Confidentiality for Collaborative Learning with SGXcited by 14
  16. 2022Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Controlcited by 13
  17. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  18. 2022When Does Re-initialization Work?cited by 12
  19. 2022Privacy-Preserving Domain Adaptation of Semantic Parserscited by 12
  20. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  21. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  22. 2022Regularising for invariance to data augmentation improves supervised learningcited by 9
  23. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  24. 2023Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learningcited by 9
  25. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  26. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 8
  27. 2024On the Performance Analysis of Momentum Method: A Frequency Domain Perspectivecited by 8
  28. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  29. 2023Conditional independence testing under misspecified inductive biasescited by 7
  30. 2023A Coefficient Makes SVRG Effectivecited by 7
  31. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 7
  32. 2021Deep Visual Navigation under Partial Observabilitycited by 6
  33. 2022Quantized Training of Gradient Boosting Decision Treescited by 6
  34. 2022On uniform-in-time diffusion approximation for stochastic gradient descentcited by 5
  35. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
  36. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  37. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  38. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4

50 most cited of 81 All 81 (JSON)