mlsanityData to 7 Oct 2026

On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)

arXiv 2102.12470 ↗First posted 24 Feb 2021cs.LG stat.MLcited by 73

Not on the map. Why?

Abstract (arXiv)

It is generally recognized that finite learning rate (LR), in contrast to infinitesimal LR, is important for good generalization in real-life deep nets. Most attempted explanations propose approximating finite-LR SGD with Ito Stochastic Differential Equations (SDEs), but formal justification for this approximation (e.g., (Li et al., 2019)) only applies to SGD with tiny LR. Experimental verification of the approximation appears computationally infeasible. The current paper clarifies the picture with the following contributions: (a) An efficient simulation algorithm SVAG that provably converges to the conventionally used Ito SDE approximation. (b) A theoretically motivated testable necessary condition for the SDE approximation and its most famous implication, the linear scaling rule (Goyal et al., 2017), to hold. (c) Experiments using this simulation to demonstrate that the previously proposed SDE approximation can meaningfully capture the training and generalization properties of common deep nets.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

39 of 57 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2018Group Normalizationcited by 1,514
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  11. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 462
27 more
  1. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  2. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  3. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  4. 2017How to Escape Saddle Points Efficientlycited by 284
  5. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  6. 2017Three Factors Influencing Minima in SGDcited by 270
  7. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  8. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  9. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  10. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  11. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  12. 2018Understanding Batch Normalizationcited by 157
  13. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  14. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  15. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  16. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  17. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  18. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  19. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 93
  20. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  21. 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
  22. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  23. 2018Fix your classifier: the marginal value of training the last weight layercited by 43
  24. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  25. 2019Sharp Analysis for Nonconvex SGD Escaping from Saddle Pointscited by 40
  26. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  27. 2020On Learning Rates and Schr"odinger Operatorscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 13; 2023: 18; 2024: 17; 2025: 11; 2026: 7.

73 papers.

Sort:

By area:

  1. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  2. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  3. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  4. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  5. 2022PAC-Bayes Compression Bounds So Tight That They Can Explain Generalizationcited by 75
  6. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  7. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  8. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  9. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  10. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  11. 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 50
  12. 2024On Scaling Up 3D Gaussian Splatting Trainingcited by 49
38 more
  1. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  2. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  3. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  4. 2023How to Scale Your EMAcited by 22
  5. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  6. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  7. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  8. 2025Critical Batch Size Revisited: A Simple Empirical Approach to Large-Batch Language Model Trainingcited by 20
  9. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  10. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  11. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  12. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  13. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  14. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  15. 2022On the Overlooked Structure of Stochastic Gradientscited by 13
  16. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  17. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  18. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  19. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  20. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  21. 2021Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equationscited by 8
  22. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  23. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  24. 2023Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descentcited by 7
  25. 2023REALM: Robust Entropy Adaptive Loss Minimization for Improved Single-Sample Test-Time Adaptationcited by 7
  26. 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
  27. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  28. 2024SDEs for Minimax Optimizationcited by 6
  29. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  30. 2022Generalization Bounds for Gradient Methods via Discrete and Continuous Priorcited by 5
  31. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  32. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  33. 2025Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Schedulingcited by 5
  34. 2022Memory-Efficient Backpropagation through Large Linear Layerscited by 4
  35. 2024Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspectivecited by 4
  36. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 4
  37. 2023Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problemscited by 3
  38. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3

50 most cited of 73 All 73 (JSON)