mlsanityData to 7 Oct 2026

On the Origin of Implicit Regularization in Stochastic Gradient Descent

arXiv 2101.12176 ↗First posted 28 Jan 2021cs.LG stat.MLcited by 183

Not on the map. Why?

Abstract (arXiv)

For infinitesimal learning rates, stochastic gradient descent (SGD) follows the path of gradient flow on the full batch loss function. However moderately large learning rates can achieve higher test accuracies, and this generalization benefit is not explained by convergence bounds, since the learning rate which maximizes test accuracy is often larger than the learning rate which minimizes training loss. To interpret this phenomenon we prove that for SGD with random shuffling, the mean SGD iterate also stays close to the path of gradient flow if the learning rate is small and finite, but on a modified loss. This modified loss is composed of the original loss function and an implicit regularizer, which penalizes the norms of the minibatch gradients. Under mild assumptions, when the batch size is small the scale of the implicit regularization term is proportional to the ratio of the learning rate to the batch size. We verify empirically that explicitly including the implicit regularizer in the loss can enhance the test accuracy when the learning rate is small.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 28 Jan 2021.

Builds on

24 of 33 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2016Wide Residual Networkscited by 3,057
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  7. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  8. 2017Three Factors Influencing Minima in SGDcited by 270
  9. 2018An Empirical Model of Large-Batch Trainingcited by 268
  10. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  11. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  12. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
12 more
  1. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  2. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  3. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  4. 2020Implicit Gradient Regularizationcited by 145
  5. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  6. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  7. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  8. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  9. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  10. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 47
  11. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  12. 2019The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 37; 2023: 40; 2024: 28; 2025: 24; 2026: 25.

183 papers.

Sort:

By area:

  1. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  3. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  4. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  5. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  6. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  7. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  8. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  9. 2021Manipulating SGD with Data Ordering Attackscited by 73
  10. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  11. 2023Can Neural Network Memorization Be Localized?cited by 66
  12. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
38 more
  1. 2025Whoever Started the Interference Should End It: Guiding Data-Free Model Merging via Task Vectorscited by 59
  2. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  3. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  4. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  5. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  6. 2021Noisy Recurrent Neural Networkscited by 42
  7. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  8. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  9. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  10. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  11. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 36
  12. 2023Deep networks for system identification: a Surveycited by 35
  13. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  14. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  15. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  16. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  17. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  18. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  19. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  20. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  21. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  22. 2023On the Implicit Bias of Adamcited by 25
  23. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  24. 2022Implicit regularization of dropoutcited by 24
  25. 2022Server-Side Stepsizes and Sampling Without Replacement Provably Help in Federated Optimizationcited by 23
  26. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  27. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  28. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  29. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  30. 2022NoisyMix: Boosting Model Robustness to Common Corruptionscited by 17
  31. 2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
  32. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  33. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  34. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  35. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  36. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  37. 2023Latent State Models of Training Dynamicscited by 15
  38. 2023Regularization and Optimal Multiclass Learningcited by 15

50 most cited of 183 All 183 (JSON)