mlsanityData to 7 Oct 2026

Fluctuation-dissipation relations for stochastic gradient descent

arXiv 1810.00004 ↗First posted 28 Sep 2018stat.ML cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 6; 2023: 6; 2024: 6; 2025: 7; 2026: 9.

47 papers.

Sort:

  1. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  2. 2021Strength of Minibatch Noise in SGDcited by 39
  3. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  4. 2022Implicit regularization of dropoutcited by 24
  5. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  6. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  7. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  8. 2026Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioningcited by 15
  9. 2021SVRG Meets AdaGrad: Painless Variance Reductioncited by 13
  10. 2021How to decay your learning ratecited by 11
  11. 2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
  12. 2021SGD Implicitly Regularizes Generalization Errorcited by 10
35 more
  1. 2022Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimizationcited by 10
  2. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
  3. 2024Formation of Representations in Neural Networkscited by 10
  4. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8
  5. 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 8
  6. 2024Stochastic weight matrix dynamics during learning and Dyson Brownian motioncited by 7
  7. 2022Optimization Planning for 3D ConvNetscited by 5
  8. 2023Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Networkcited by 5
  9. 2023Machine learning in and out of equilibriumcited by 5
  10. 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
  11. 2021Fluctuation-dissipation Type Theorem in Stochastic Linear Learningcited by 3
  12. 2021Revisiting the Effects of Stochasticity for Hamiltonian Samplerscited by 3
  13. 2023Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problemscited by 3
  14. 2021Generative Adversarial Network: Some Analytical Perspectivescited by 2
  15. 2021Shift-Curvature, SGD, and Generalizationcited by 2
  16. 2021Multilayer Lookahead: a Nested Version of Lookaheadcited by 2
  17. 2023Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directionscited by 2
  18. 2025AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descentcited by 1
  19. 2025Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networkscited by 1
  20. 2025Closed-form ℓr norm scaling with data for overparameterized linear regression and diagonal linear networks under ℓp biascited by 1
  21. 2025Contribution of task-irrelevant stimuli to drift of neural representationscited by 1
  22. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 1
  23. 2022Studying Generalization Through Data Averagingcited by 0
  24. 2023Climate Intervention Analysis using AI Model Guided by Statistical Physics Principlescited by 0
  25. 2024Dynamics of Supervised and Reinforcement Learning in the Non-Linear Perceptroncited by 0
  26. 2024Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descentcited by 0
  27. 2024Random Matrix Theory for Stochastic Gradient Descentcited by 0
  28. 2025The Universe Learning Itself: On the Evolution of Dynamics from the Big Bang to Machine Intelligencecited by 0
  29. 2026Thermodynamics of Reinforcement Learning Curriculacited by 0
  30. 2026Nonreversible Gauge Fields in Fokker--Planck Dynamics: Supersymmetric Hamiltonians and Learned Finite Forcescited by 0
  31. 2026The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Modelscited by 0
  32. 2026Speed Limit for Information Acquisition in Stochastic Learning Dynamicscited by 0
  33. 2026An Analytical Theory of Auxiliary Learningcited by 0
  34. 2026Optimizer-dependent training dynamics converge to the same one-third optimal data scalingcited by 0
  35. 2026Learning the identity: a case study of how SGD selects among functional decompositionscited by 0