mlsanityData to 14 Oct 2026

Stochastic modified equations and adaptive stochastic gradient algorithms

arXiv 1511.06251 ↗First posted 19 Nov 2015cs.LG stat.MLcited by 142

Not on the map. Why?

Abstract (arXiv)

We develop the method of stochastic modified equations (SME), in which stochastic gradient algorithms are approximated in the weak sense by continuous-time stochastic differential equations. We exploit the continuous formulation together with optimal control theory to derive novel adaptive hyper-parameter adjustment policies. Our algorithms have competitive performance with the added benefit of being robust to varying models and datasets. This provides a general methodology for the analysis and design of stochastic gradient algorithms.

Key facts. Cited by 142 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 34; 2022: 13; 2023: 22; 2024: 21; 2025: 20; 2026: 32.

142 papers.

Sort:

By area:

  1. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  2. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  3. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  4. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  5. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  6. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  7. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  8. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  9. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
  10. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  11. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 36
  12. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 35
38 more
  1. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  2. 2021Power-law escape rate of SGDcited by 29
  3. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 29
  4. 2021Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalizationcited by 28
  5. 2023On the Implicit Bias of Adamcited by 27
  6. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 26
  7. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  8. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  9. 2021Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcited by 23
  10. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  11. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22
  12. 2025A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedulescited by 22
  13. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  14. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  15. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  16. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  17. 2021Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Modelscited by 16
  18. 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
  19. 2021Discretization Drift in Two-Player Gamescited by 14
  20. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  21. 2026A Defense of the Quadratic Modelcited by 14
  22. 2021Acceleration via Fractal Learning Rate Schedulescited by 13
  23. 2021Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterizationcited by 13
  24. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 13
  25. 2021Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injectionscited by 12
  26. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  27. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  28. 2023Stochastic Modified Equations and Dynamics of Dropout Algorithmcited by 11
  29. 2021Edge of chaos as a guiding principle for modern neural network trainingcited by 10
  30. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  31. 2026The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspectivecited by 10
  32. 2026Momentum Further Constrains Sharpness at the Edge of Stochastic Stabilitycited by 10
  33. 2021Personalized Algorithm Generation: A Case Study in Learning ODE Integratorscited by 9
  34. 2021On the Hyperparameters in Stochastic Gradient Descent with Momentumcited by 9
  35. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  36. 2026Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stabilitycited by 9
  37. 2021Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equationscited by 8
  38. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8

50 most cited of 142 All 142 (JSON)