mlsanityData to 7 Oct 2026

A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent

arXiv 2310.00692 ↗First posted 1 Oct 2023cs.LG stat.MLcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we provide a theoretical study of noise geometry for minibatch stochastic gradient descent (SGD), a phenomenon where noise aligns favorably with the geometry of local landscape. We propose two metrics, derived from analyzing how noise influences the loss and subspace projection dynamics, to quantify the alignment strength. We show that for (over-parameterized) linear models and two-layer nonlinear networks, when measured by these metrics, the alignment can be provably guaranteed under conditions independent of the degree of over-parameterization. To showcase the utility of our noise geometry characterizations, we present a refined analysis of the mechanism by which SGD escapes from sharp minima. We reveal that unlike gradient descent (GD), which escapes along the sharpest directions, SGD tends to escape from flatter directions and cyclical learning rates can exploit this SGD characteristic to navigate more effectively towards flatter regions. Lastly, extensive experiments are provided to support our theoretical findings.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 38 of its 49 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

38 of 49 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  7. 2019Introduction to Online Convex Optimizationcited by 793
  8. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  11. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  12. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
26 more
  1. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  2. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  3. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  4. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  5. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  6. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  7. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  8. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  9. 2022Understanding the unstable convergence of gradient descentcited by 84
  10. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  11. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  12. 2018Escaping Saddles with Stochastic Gradientscited by 66
  13. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  14. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 60
  15. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  16. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  17. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  18. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  19. 2021Strength of Minibatch Noise in SGDcited by 39
  20. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  21. 2019Bootstrapping Upper Confidence Boundcited by 36
  22. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 36
  23. 2018On exponential convergence of SGD in non-convex over-parametrized learningcited by 33
  24. 2021Power-law escape rate of SGDcited by 29
  25. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  26. 2023Aiming towards the minimizers: fast convergence of SGD for overparametrized problemscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 4.

8 papers.

  1. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 6
  2. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 4
  3. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 4
  4. 2026Why Muon Outperforms Adam: A Curvature Perspectivecited by 4
  5. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  6. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  7. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  8. 2026Marginals Before Conditionalscited by 0