# Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning

arXiv: https://arxiv.org/abs/1803.01927 · HTML: https://mlsanity.com/paper/1803.01927/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Mar 2018.

## Built on by (most cited first, 20 of 20)

- [The Local Learning Coefficient: A Singularity-Aware Complexity Measure](https://mlsanity.com/paper/2308.12108/) (2023)
- [Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition](https://mlsanity.com/paper/2310.06301/) (2023)
- [Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions](https://mlsanity.com/paper/2206.01246/) (2022)
- [Information Complexity and Generalization Bounds](https://mlsanity.com/paper/2105.01747/) (2021)
- [Why flatness does and does not correlate with generalization for deep neural networks](https://mlsanity.com/paper/2103.06219/) (2021)
- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [On the different regimes of Stochastic Gradient Descent](https://mlsanity.com/paper/2309.10688/) (2023)
- [SGD Implicitly Regularizes Generalization Error](https://mlsanity.com/paper/2104.04874/) (2021)
- [Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning](https://mlsanity.com/paper/2301.13703/) (2023)
- [Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise](https://mlsanity.com/paper/2312.14567/) (2023)
- [The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization](https://mlsanity.com/paper/2409.09745/) (2024)
- [Stochastic Gradient Descent-Induced Drift of Representation in a Two-Layer Neural Network](https://mlsanity.com/paper/2302.02563/) (2023)
- [Estimating the Local Learning Coefficient at Scale](https://mlsanity.com/paper/2402.03698/) (2024)
- [Phases of learning dynamics in artificial neural networks: with or without mislabeled data](https://mlsanity.com/paper/2101.06509/) (2021)
- [SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training](https://mlsanity.com/paper/2505.23489/) (2025)
- [Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent](https://mlsanity.com/paper/2112.01433/) (2021)
- [PoF: Post-Training of Feature Extractor for Improving Generalization](https://mlsanity.com/paper/2207.01847/) (2022)
- [Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation](https://mlsanity.com/paper/2311.14120/) (2023)
- [Physics Inspired Approaches To Understanding Gaussian Processes](https://mlsanity.com/paper/2305.10748/) (2023)
- [Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?](https://mlsanity.com/paper/2511.07308/) (2025)

## Abstract (arXiv)

Finding parameters that minimise a loss function is at the core of many machine learning methods. The Stochastic Gradient Descent algorithm is widely used and delivers state of the art results for many problems. Nonetheless, Stochastic Gradient Descent typically cannot find the global minimum, thus its empirical effectiveness is hitherto mysterious. We derive a correspondence between parameter inference and free energy minimisation in statistical physics. The degree of undersampling plays the role of temperature. Analogous to the energy-entropy competition in statistical physics, wide but shallow minima can be optimal if the system is undersampled, as is typical in many applications. Moreover, we show that the stochasticity in the algorithm has a non-trivial correlation structure which systematically biases it towards wide minima. We illustrate our argument with two prototypical models: image classification using deep learning, and a linear neural network where we can analytically reveal the relationship between entropy and out-of-sample error.