mlsanityData to 7 Oct 2026

Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

arXiv 2605.22644 ↗First posted 21 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion. However, this approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate. In this work, we propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling. Starting directly from the discrete update, we derive a master equation for the parameter distribution and obtain a discrete Fokker--Planck equation that differs from the standard Langevin form at order eta^2. Using this framework, we analyze SGD dynamics near critical points of the loss. We show that the behavior decomposes along the eigenbasis of the mean Hessian into qualitatively distinct regimes. In particular, nearly-flat directions do not admit a stationary distribution: the variance grows over time, corresponding to effective diffusion along valleys with a coefficient proportional to the learning rate. We provide empirical evidence supporting these predictions on neural network models in computer vision and natural language processing, observing a clear qualitative separation between confined and diffusive modes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

18 of 31 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  4. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  5. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 141
  6. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  7. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  8. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  9. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 85
  10. 2015Why Random Reshuffling Beats Stochastic Gradient Descentcited by 78
  11. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  12. 2018Random Shuffling Beats SGD after Finite Epochscited by 66
6 more
  1. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  2. 2020Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noisecited by 33
  3. 2022Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutionscited by 22
  4. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  5. 2018The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effectscited by 9
  6. 2021Phases of learning dynamics in artificial neural networks: with or without mislabeled datacited by 4

Built on by

Not cited yet.