mlsanityData to 7 Oct 2026

SGD Implicitly Regularizes Generalization Error

arXiv 2104.04874 ↗First posted 10 Apr 2021cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We discuss further improvements to these calculations and comment on possible implications for stochastic optimization.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 10 Apr 2021.

Builds on

17 of 26 references.

References · 17

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  7. 2017Three Factors Influencing Minima in SGDcited by 270
  8. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  9. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  10. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  11. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  12. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
5 more
  1. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  2. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  3. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  4. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  5. 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 2; 2023: 2; 2024: 1; 2025: 2.

10 papers.

  1. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  2. 2021On the benefits of representation regularization in invariance based domain generalizationcited by 17
  3. 2023Implicit meta-learning may lead language models to trust more reliable sourcescited by 9
  4. 2022Rethinking Sharpness-Aware Minimization as Variational Inferencecited by 8
  5. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  6. 2022Noise Injection as a Probe of Deep Learning Dynamicscited by 2
  7. 2023Transgressing the boundaries: towards a rigorous understanding of deep learning and its (non-)robustnesscited by 1
  8. 2021Generalization by design: Shortcuts to Generalization in Deep Learningcited by 0
  9. 2025Overfitting has a limitation: a model-independent generalization gap bound based on R'enyi entropycited by 0
  10. 2025Tight Generalization Error Bounds for Stochastic Gradient Descent in Non-convex Learningcited by 0