mlsanityData to 7 Oct 2026

Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundations

arXiv 1811.01558 ↗First posted 5 Nov 2018cs.LG stat.MLcited by 109

Not on the map. Why?

Abstract (arXiv)

We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood mathematically as an weak approximation, which leads to a number of precise and useful results on the approximations of stochastic gradient descent (SGD), momentum SGD and stochastic Nesterov's accelerated gradient method in the general setting of stochastic objectives. We also demonstrate through explicit calculations that this continuous-time approach can uncover important analytical insights into the stochastic gradient algorithms under consideration that may not be easy to obtain in a purely discrete-time setting.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 13; 2023: 23; 2024: 16; 2025: 18; 2026: 24.

109 papers.

Sort:

By area:

  1. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  2. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  3. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  4. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  5. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  6. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  7. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  8. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  9. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  10. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  11. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  12. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
38 more
  1. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  2. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  3. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  4. 2023How to Scale Your EMAcited by 22
  5. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 22
  6. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  7. 2024Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descentcited by 19
  8. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  9. 2023The Marginal Value of Momentum for Small Learning Rate SGDcited by 16
  10. 2021Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Modelscited by 15
  11. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  12. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  13. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  14. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  15. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  16. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  17. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  18. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  19. 2021On the Hyperparameters in Stochastic Gradient Descent with Momentumcited by 9
  20. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  21. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 8
  22. 2021A Continuous-time Stochastic Gradient Descent Method for Continuous Datacited by 7
  23. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 7
  24. 2023Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descentcited by 7
  25. 2023Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Biascited by 7
  26. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 7
  27. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  28. 2021Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field trainingcited by 5
  29. 2022On uniform-in-time diffusion approximation for stochastic gradient descentcited by 5
  30. 2023High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariancecited by 5
  31. 2023Backward error analysis and the qualitative behaviour of stochastic optimization algorithms: Application to stochastic coordinate descentcited by 5
  32. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  33. 2024Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effectscited by 5
  34. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 4
  35. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  36. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 4
  37. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 4
  38. 2021Adversarial Training for Gradient Descent: Analysis Through its Continuous-time Approximationcited by 3

50 most cited of 109 All 109 (JSON)