mlsanityData to 7 Oct 2026

Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks

arXiv 1710.11029 ↗First posted 30 Oct 2017cs.LG cond-mat.stat-mech math.OCcited by 108

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent (SGD) is widely believed to perform implicit regularization when used to train deep neural networks, but the precise manner in which this occurs has thus far been elusive. We prove that SGD minimizes an average potential over the posterior distribution of weights along with an entropic regularization term. This potential is however not the original loss function in general. So SGD does perform variational inference, but for a different loss than the one used to compute the gradients. Even more surprisingly, SGD does not even converge in the classical sense: we show that the most likely trajectories of SGD for deep networks do not behave like Brownian motion around critical points. Instead, they resemble closed loops with deterministic components. We prove that such "out-of-equilibrium" behavior is a consequence of highly non-isotropic gradient noise in SGD; the covariance matrix of mini-batch gradients for deep networks has a rank as small as 1% of its dimension. We provide extensive empirical validation of these claims, proven in the appendix.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Oct 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 18; 2023: 24; 2024: 8; 2025: 13; 2026: 17.

108 papers.

Sort:

By area:

  1. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  2. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  3. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  4. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  5. 2021The Bayesian Learning Rulecited by 95
  6. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  7. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  8. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  9. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  10. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  11. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  12. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
38 more
  1. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  2. 2021Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Errorcited by 32
  3. 2024On-the-fly Modulation for Balanced Multimodal Learningcited by 32
  4. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  5. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  6. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  7. 2022Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometrycited by 27
  8. 2025SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Trainingcited by 27
  9. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  10. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  11. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  12. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20
  13. 2023The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifoldcited by 20
  14. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  15. 2021Lifelong Learning with Sketched Structural Regularizationcited by 15
  16. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  17. 2025FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Modelscited by 15
  18. 2021Does the Data Induce Capacity Control in Deep Learning?cited by 13
  19. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  20. 2022The activity-weight duality in feed forward neural networks: The geometric determinants of generalizationcited by 12
  21. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  22. 2021SGD Implicitly Regularizes Generalization Errorcited by 10
  23. 2021On the Hyperparameters in Stochastic Gradient Descent with Momentumcited by 9
  24. 2022Predicting the generalization gap in neural networks using topological data analysiscited by 9
  25. 2022On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walkscited by 9
  26. 2023Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learningcited by 9
  27. 2023Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noisecited by 9
  28. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  29. 2021Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equationscited by 8
  30. 2023A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustnesscited by 7
  31. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  32. 2024Querying Easily Flip-flopped Samples for Deep Active Learningcited by 7
  33. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  34. 2022A picture of the space of typical learnable taskscited by 6
  35. 2021On Large Batch Training and Sharp Minima: A Fokker-Planck Perspectivecited by 5
  36. 2022Accelerating Self-Supervised Learning via Efficient Training Strategiescited by 5
  37. 2023Machine learning in and out of equilibriumcited by 5
  38. 2021Phases of learning dynamics in artificial neural networks: with or without mislabeled datacited by 4

50 most cited of 108 All 108 (JSON)