# Fluctuation-dissipation relations for stochastic gradient descent

arXiv: https://arxiv.org/abs/1810.00004 · HTML: https://mlsanity.com/paper/1810.00004/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2018.

## Built on by (most cited first, 30 of 48)

- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [Strength of Minibatch Noise in SGD](https://mlsanity.com/paper/2102.05375/) (2021)
- [SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality](https://mlsanity.com/paper/2102.04396/) (2021)
- [Implicit regularization of dropout](https://mlsanity.com/paper/2207.05952/) (2022)
- [Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms](https://mlsanity.com/paper/2106.04881/) (2021)
- [Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions](https://mlsanity.com/paper/2206.07252/) (2022)
- [The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion](https://mlsanity.com/paper/2107.09133/) (2021)
- [Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning](https://mlsanity.com/paper/2603.09697/) (2026)
- [SVRG Meets AdaGrad: Painless Variance Reduction](https://mlsanity.com/paper/2102.09645/) (2021)
- [How to decay your learning rate](https://mlsanity.com/paper/2103.12682/) (2021)
- [On a continuous time model of gradient descent dynamics and instability in deep learning](https://mlsanity.com/paper/2302.01952/) (2023)
- [SGD Implicitly Regularizes Generalization Error](https://mlsanity.com/paper/2104.04874/) (2021)
- [Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization](https://mlsanity.com/paper/2201.11137/) (2022)
- [Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learning](https://mlsanity.com/paper/2210.00690/) (2022)
- [Formation of Representations in Neural Networks](https://mlsanity.com/paper/2410.03006/) (2024)
- [Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning](https://mlsanity.com/paper/2505.12387/) (2025)
- [How Neural Networks Learn the Support is an Implicit Regularization Effect of SGD](https://mlsanity.com/paper/2406.11110/) (2024)
- [Stochastic weight matrix dynamics during learning and Dyson Brownian motion](https://mlsanity.com/paper/2407.16427/) (2024)
- [Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network](https://mlsanity.com/paper/2302.02563/) (2023)
- [Optimization Planning for 3D ConvNets](https://mlsanity.com/paper/2201.04021/) (2022)
- [Machine learning in and out of equilibrium](https://mlsanity.com/paper/2306.03521/) (2023)
- [SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training](https://mlsanity.com/paper/2505.23489/) (2025)
- [Fluctuation-dissipation Type Theorem in Stochastic Linear Learning](https://mlsanity.com/paper/2106.02220/) (2021)
- [Revisiting the Effects of Stochasticity for Hamiltonian Samplers](https://mlsanity.com/paper/2106.16200/) (2021)
- [Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems](https://mlsanity.com/paper/2309.05337/) (2023)
- [Generative Adversarial Network: Some Analytical Perspectives](https://mlsanity.com/paper/2104.12210/) (2021)
- [Shift-Curvature, SGD, and Generalization](https://mlsanity.com/paper/2108.09507/) (2021)
- [Multilayer Lookahead: a Nested Version of Lookahead](https://mlsanity.com/paper/2110.14254/) (2021)
- [Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directions](https://mlsanity.com/paper/2306.05300/) (2023)
- [AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent](https://mlsanity.com/paper/2505.21651/) (2025)

## Abstract (arXiv)

The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics. In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity. Here, we derive stationary fluctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm. These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule. We can further use the relations to efficiently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity. Our claims are empirically verified.