# What is the long-run distribution of stochastic gradient descent? A large deviations analysis

arXiv: https://arxiv.org/abs/2406.09241 · HTML: https://mlsanity.com/paper/2406.09241/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2024.

## Built on by (most cited first, 17 of 17)

- [Large Deviation Upper Bounds and Improved MSE Rates of Nonlinear SGD: Heavy-tailed Noise and Power of Symmetry](https://mlsanity.com/paper/2410.15637/) (2024)
- [Paths and Ambient Spaces in Neural Loss Landscapes](https://mlsanity.com/paper/2503.03382/) (2025)
- [Variations on the Expectation due to Changes in the Probability Measure](https://mlsanity.com/paper/2502.02887/) (2025)
- [Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processes](https://mlsanity.com/paper/2505.19087/) (2025)
- [Decentralized Machine Learning with Centralized Performance Guarantees via Gibbs Algorithms](https://mlsanity.com/paper/2604.20492/) (2026)
- [Tight Long-Term Tail Decay of (Clipped) SGD in Non-Convex Optimization](https://mlsanity.com/paper/2602.05657/) (2026)
- [Large Spikes in Stochastic Gradient Descent: A Large-Deviations View](https://mlsanity.com/paper/2603.10079/) (2026)
- [Shuffling the Data, Stretching the Step-size: Sharper Bias in constant step-size SGD](https://mlsanity.com/paper/2604.10373/) (2026)
- [An Exponential Averaging Process with Strong Convergence Properties](https://mlsanity.com/paper/2505.10605/) (2025)
- [Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD](https://mlsanity.com/paper/2505.18535/) (2025)
- [Multi-agent learning under uncertainty: Recurrence vs. concentration](https://mlsanity.com/paper/2512.08132/) (2025)
- [The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weights](https://mlsanity.com/paper/2512.10188/) (2025)
- [Proximal basin hopping: global optimization with guarantees](https://mlsanity.com/paper/2605.18364/) (2026)
- [Adynamical systems view of training generativemodels and the memorization phenomenon](https://mlsanity.com/paper/2605.19483/) (2026)
- [Equivalence Between Nested Gibbs Measures and Log-Linear Combinations of Gibbs Measures](https://mlsanity.com/paper/2609.19988/) (2026)
- [Machine Unlearning for Gibbs Supervised Learning Algorithms](https://mlsanity.com/paper/2609.29409/) (2026)
- [Distribution of hitting times for dissipative random dynamical systems on ℝd, with application to stochastic gradient descent](https://mlsanity.com/paper/2609.30274/) (2026)

## Abstract (arXiv)

In this paper, we examine the long-run distribution of stochastic gradient descent (SGD) in general, non-convex problems. Specifically, we seek to understand which regions of the problem's state space are more likely to be visited by SGD, and by how much. Using an approach based on the theory of large deviations and randomly perturbed dynamical systems, we show that the long-run distribution of SGD resembles the Boltzmann-Gibbs distribution of equilibrium thermodynamics with temperature equal to the method's step-size and energy levels determined by the problem's objective and the statistics of the noise. In particular, we show that, in the long run, (a) the problem's critical region is visited exponentially more often than any non-critical region; (b) the iterates of SGD are exponentially concentrated around the problem's minimum energy state (which does not always coincide with the global minimum of the objective); (c) all other connected components of critical points are visited with frequency that is exponentially proportional to their energy level; and, finally (d) any component of local maximizers or saddle points is "dominated" by a component of local minimizers which is visited exponentially more often.