mlsanityData to 14 Oct 2026

SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training

arXiv 2505.23489 ↗First posted 29 May 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We present a thermodynamic interpretation of the stationary behavior of stochastic gradient descent (SGD) under fixed learning rates (LRs) in neural network training. We show that SGD implicitly minimizes a free energy function F=U-TS, balancing training loss U and the entropy of the weights distribution S, with temperature T determined by the LR. This perspective offers a new lens on why high LRs prevent training from converging to the loss minima and how different LRs lead to stabilization at different loss levels. We empirically validate the free energy framework on both underparameterized (UP) and overparameterized (OP) models. UP models consistently follow free energy minimization, with temperature increasing monotonically with LR, while for OP models, the temperature effectively drops to zero at low LRs, causing SGD to minimize the loss directly and converge to an optimum. We attribute this mismatch to differences in the signal-to-noise ratio of stochastic gradients near optima, supported by both a toy example and neural network experiments.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

55 of 69 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  9. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  10. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  11. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  12. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
38 more
  1. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  2. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  4. 2023Symbolic Discovery of Optimization Algorithmscited by 434
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  7. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  8. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  9. 2017Three Factors Influencing Minima in SGDcited by 273
  10. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  11. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  12. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  13. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  14. 2020Implicit Gradient Regularizationcited by 146
  15. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  16. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  17. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 124
  18. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  19. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  20. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  21. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  22. 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
  23. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  24. 2018Fluctuation-dissipation relations for stochastic gradient descentcited by 48
  25. 2020Understanding Why Neural Networks Generalize Well Through GSNR of Parameterscited by 40
  26. 2021Strength of Minibatch Noise in SGDcited by 39
  27. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 39
  28. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 37
  29. 2024Towards Few-Shot Adaptation of Foundation Models via Multitask Finetuningcited by 31
  30. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  31. 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20
  32. 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
  33. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  34. 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
  35. 2023On the different regimes of Stochastic Gradient Descentcited by 12
  36. 2024Abrupt Learning in Transformers: A Case Study on Matrix Completioncited by 11
  37. 2018Fixed-Time Stable Gradient Flows: Applications to Continuous-Time Optimizationcited by 10
  38. 2018TherML: Thermodynamics of Machine Learningcited by 9

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  2. 2026Geodesic Gradient Descent: A Generic and Learning-rate-free Optimizer on Objective Function-induced Manifoldscited by 1
  3. 2025Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?cited by 0
  4. 2026Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networkscited by 0