mlsanityData to 14 Oct 2026

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning

arXiv 2306.08590 ↗First posted 14 Jun 2023cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

The success of SGD in deep learning has been ascribed by prior works to the implicit bias induced by finite batch sizes ("SGD noise"). While prior works focused on offline learning (i.e., multiple-epoch training), we study the impact of SGD noise on online (i.e., single epoch) learning. Through an extensive empirical analysis of image and language data, we demonstrate that small batch sizes do not confer any implicit bias advantages in online learning. In contrast to offline learning, the benefits of SGD noise in online learning are strictly computational, facilitating more cost-effective gradient steps. This suggests that SGD in the online regime can be construed as taking noisy steps along the "golden path" of the noiseless gradient descent algorithm. We study this hypothesis and provide supporting evidence in loss and function space. Our findings challenge the prevailing understanding of SGD and offer novel insights into its role in online learning.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

42 of 56 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  10. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  11. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
30 more
  1. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  2. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
  3. 2017Three Factors Influencing Minima in SGDcited by 273
  4. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  5. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  6. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  7. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  8. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  9. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  10. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  11. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  12. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  13. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  14. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  15. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 102
  16. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  17. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  18. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  19. 2018A Walk with SGDcited by 78
  20. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  21. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  22. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 62
  23. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 59
  24. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 56
  25. 2019One Epoch Is All You Needcited by 41
  26. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 27
  27. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  28. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 22
  29. 2021A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizescited by 20
  30. 2022Disentangling the Mechanisms Behind Implicit Regularization in SGDcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 3.

10 papers.

  1. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  2. 2025Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wastefulcited by 41
  3. 2024Rethinking Conventional Wisdom in Machine Learning: From Generalization to Scalingcited by 23
  4. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  5. 2024Time Transfer: On Optimal Learning Rate and Batch Size In The Infinite Data Limitcited by 9
  6. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  7. 2026When Do Larger Batches Help Scale LLM Reinforcement Learning?cited by 2
  8. 2025Zero Generalization Error Theorem for Random Interpolators via Algebraic Geometrycited by 0
  9. 2026Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networkscited by 0
  10. 2026Why Does Adaptive Batching Help LLM Pretraining? A Perspective from Unbounded Variancecited by 0