mlsanityData to 14 Oct 2026

Stochastic Modified Flows, Mean-Field Limits and Dynamics of Stochastic Gradient Descent

arXiv 2302.07125 ↗First posted 14 Feb 2023math.PR cs.LG math.APcited by 7

Not on the map. Why?

Abstract (arXiv)

We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 18 of its 50 references that are in the map. First posted on arXiv 14 Feb 2023.

Builds on

18 of 50 references.

References · 18

  1. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  3. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  4. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  5. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  6. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  7. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  8. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  9. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  10. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  11. 2019Machine Learning from a Continuous Viewpointcited by 50
  12. 2019Analysis of a Two-Layer Neural Network via Displacement Convexitycited by 27
6 more
  1. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  2. 2022Uniform-in-time propagation of chaos for mean field Langevin dynamicscited by 23
  3. 2019Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximationcited by 15
  4. 2023Exponential convergence rates for momentum stochastic gradient descent in the overparametrized settingcited by 8
  5. 2022On uniform-in-time diffusion approximation for stochastic gradient descentcited by 6
  6. 2020Noise-induced degeneration in online learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 3; 2026: 1.

7 papers.

  1. 2025ODE approximation for the Adam algorithm: General and overparametrized settingcited by 10
  2. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 8
  3. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  4. 2023Unlocking optimal batch size schedules using continuous-time control and perturbation theorycited by 3
  5. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  6. 2025Effective continuous equations for adaptive SGD: a stochastic analysis viewcited by 1
  7. 2026A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networkscited by 0