mlsanityData to 7 Oct 2026

Revisiting the Noise Model of Stochastic Gradient Descent

arXiv 2303.02749 ↗First posted 5 Mar 2023cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

The stochastic gradient noise (SGN) is a significant factor in the success of stochastic gradient descent (SGD). Following the central limit theorem, SGN was initially modeled as Gaussian, and lately, it has been suggested that stochastic gradient noise is better characterized using Sα S L'evy distribution. This claim was allegedly refuted and rebounded to the previously suggested Gaussian noise model. This paper presents solid, detailed empirical evidence that SGN is heavy-tailed and better depicted by the Sα S distribution. Furthermore, we argue that different parameters in a deep neural network (DNN) hold distinct SGN characteristics throughout training. To more accurately approximate the dynamics of SGD near a local minimum, we construct a novel framework in ℝN, based on L'evy-driven stochastic differential equation (SDE), where one-dimensional L'evy processes model each parameter in the DNN. Next, we show that SGN jump intensity (frequency and amplitude) depends on the learning rate decay mechanism (LRdecay); furthermore, we demonstrate empirically that the LRdecay effect may stem from the reduction of the SGN and not the decrease in the step size. Based on our analysis, we examine the mean escape time, trapping probability, and more properties of DNNs near local minima. Finally, we prove that the training process will likely exit from the basin in the direction of parameters with heavier tail SGN. We will share our code for reproducibility.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 25 of its 67 references that are in the map. First posted on arXiv 5 Mar 2023.

Builds on

25 of 67 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  7. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  8. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  9. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  10. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  11. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  12. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
13 more
  1. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  2. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  3. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  4. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  5. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  6. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  7. 2017The loss surface of deep and wide neural networkscited by 74
  8. 2017A Hitting Time Analysis of Stochastic Gradient Langevin Dynamicscited by 70
  9. 2016A Variational Analysis of Stochastic Gradient Algorithmscited by 56
  10. 2021Power-law escape rate of SGDcited by 29
  11. 2017Fractional Langevin Monte Carlo: Exploring L'evy Driven Stochastic Differential Equations for Markov Chain Monte Carlocited by 12
  12. 2020Dynamic of Stochastic Gradient Descent with State-Dependent Noisecited by 12
  13. 2019On the Noisy Gradient Descent that Generalizes as SGDcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 11; 2026: 18.

35 papers.

Sort:

  1. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  2. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
  3. 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
  4. 2025Can SGD Handle Heavy-Tailed Noise?cited by 15
  5. 2026Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decompositioncited by 8
  6. 2024Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetrycited by 7
  7. 2025ICR-RL: Deep Reinforcement Learning via In-Context Regressioncited by 5
  8. 2025TempoControl: Temporal Attention Guidance for Text-to-Video Modelscited by 5
  9. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
  10. 2025Sharp High-Probability Rates for Nonlinear SGD under Heavy-Tailed Noise via Symmetrizationcited by 3
  11. 2025Second-order Optimization under Heavy-Tailed Noise: Hessian Clipping and Sample Complexity Limitscited by 3
  12. 2026Can Adaptive Gradient Methods Converge under Heavy-Tailed Noise? A Case Study of AdaGradcited by 3
23 more
  1. 2025Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descentcited by 2
  2. 2026Optimal Asynchronous Stochastic Nonconvex Optimization under Heavy-Tailed Noisecited by 2
  3. 2026Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noisecited by 2
  4. 2026Muon Converges under Heavy-Tailed Noise: Nonconvex H"older-Smooth Empirical Risk Minimizationcited by 2
  5. 2026Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameterscited by 2
  6. 2026In-Expectation Convergence of Stochastic Gradient Methods under Heavy-Tailed Noisecited by 2
  7. 2024Distributed Stochastic Gradient Descent with Staleness: A Stochastic Delay Differential Equation Based Frameworkcited by 1
  8. 2024Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guaranteescited by 1
  9. 2025Federated Stochastic Minimax Optimization under Heavy-Tailed Noisescited by 1
  10. 2026Near-Optimal Decentralized Stochastic Nonconvex Optimization with Heavy-Tailed Noisecited by 1
  11. 2026Robust and Fast Training via Per-Sample Clippingcited by 1
  12. 2026Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Successcited by 1
  13. 2026Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?cited by 1
  14. 2026The Convergence Behavior of Adam under Heavy-Tailed Noisecited by 1
  15. 2024Effect of Random Learning Rate: Theoretical Analysis of SGD Dynamics in Non-Convex Optimization via Stationary Distributioncited by 0
  16. 2025FedEve: On Bridging the Client Drift and Period Drift for Cross-device Federated Learningcited by 0
  17. 2025Nonconvex Decentralized Stochastic Bilevel Optimization under Heavy-Tailed Noisecited by 0
  18. 2026Stability and Generalization of Stochastic Nonconvex Optimization under Heavy-Tailed Noise: From Gradient Norm to Goldstein Stationaritycited by 0
  19. 2026Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient samplingcited by 0
  20. 2026Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalizationcited by 0
  21. 2026Interleaved Noise Injection Improves Clean, Corrupted, and OOD Performancecited by 0
  22. 2026TR-SSQP: A Trust-Region Method for Constrained Stochastic Optimization under Heavy-Tailed Noisecited by 0
  23. 2026High-Probability Guarantees for SGD under β-Heavy-Tailed Gradient Noisecited by 0