mlsanityData to 7 Oct 2026

Strength of Minibatch Noise in SGD

arXiv 2102.05375 ↗First posted 10 Feb 2021cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

The noise in stochastic gradient descent (SGD), caused by minibatch sampling, is poorly understood despite its practical importance in deep learning. This work presents the first systematic study of the SGD noise and fluctuations close to a local minimum. We first analyze the SGD noise in linear regression in detail and then derive a general formula for approximating SGD noise in different types of minima. For application, our results (1) provide insight into the stability of training a neural network, (2) suggest that a large learning rate can help generalization by introducing an implicit regularization, (3) explain why the linear learning rate-batchsize scaling law fails at a large learning rate or at a small batchsize and (4) can provide an understanding of how discrete-time nature of SGD affects the recently discovered power-law phenomenon of SGD.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 25 of its 55 references that are in the map. First posted on arXiv 10 Feb 2021.

Builds on

25 of 55 references.

References · 25

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  7. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  8. 2017Three Factors Influencing Minima in SGDcited by 270
  9. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  10. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  11. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  12. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
13 more
  1. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  2. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  3. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  4. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 84
  5. 2018A Walk with SGDcited by 74
  6. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  7. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  8. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  9. 2019Understanding the Role of Momentum in Stochastic Gradient Methodscited by 42
  10. 2020Recent advances in deep learning theorycited by 41
  11. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  12. 2018Theory of Deep Learning IIb: Optimization Properties of SGDcited by 12
  13. 2020Dynamic of Stochastic Gradient Descent with State-Dependent Noisecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 11; 2024: 5; 2025: 5; 2026: 6.

39 papers.

Sort:

By area:

  1. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  2. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  3. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  4. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  5. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20
  6. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  7. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  8. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  9. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  10. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  11. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  12. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 10
27 more
  1. 2023Exact Mean Square Linear Stability Analysis for SGDcited by 10
  2. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  3. 2024Do Parameters Reveal More than Loss for Membership Inference?cited by 9
  4. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  5. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  6. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  7. 2026Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?cited by 6
  8. 2022Generalization Bounds for Gradient Methods via Discrete and Continuous Priorcited by 5
  9. 2024Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noisecited by 4
  10. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  11. 2022Stochastic Neural Networks with Infinite Width are Deterministiccited by 3
  12. 2023Approximate Heavy Tails in Offline (Multi-Pass) Stochastic Gradient Descentcited by 3
  13. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  14. 2023Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directionscited by 2
  15. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  16. 2026On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentialscited by 2
  17. 2022Noise Injection Node Regularization for Robust Learningcited by 1
  18. 2023Evolutionary Algorithms in the Light of SGD: Limit Equivalence, Minima Flatness, and Transfer Learningcited by 1
  19. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  20. 2021On the Distributional Properties of Adaptive Gradientscited by 0
  21. 2024How Learning Dynamics Drive Adversarially Robust Generalization?cited by 0
  22. 2025FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learningcited by 0
  23. 2025BOND: License to Train with Black-Box Functionscited by 0
  24. 2026Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attentioncited by 0
  25. 2026Improving Adversarial Robustness of Attribution via Implicit Regularizationcited by 0
  26. 2026Accurate Large-sample Uncertainty Quantification using Stochastic Gradient Markov Chain Monte Carlocited by 0
  27. 2026FeLoG: Scalable and Efficient Distributed Graph Embedding with Feedback Loop Mechanismcited by 0