Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?
Not on the map. Why?
Abstract (arXiv)
Understanding the training dynamics of deep neural networks remains a major open problem, with physics-inspired approaches offering promising insights. Building on this perspective, we develop a thermodynamic framework to describe the stationary distributions of stochastic gradient descent (SGD) with weight decay for scale-invariant neural networks, a setting that both reflects practical architectures with normalization layers and permits theoretical analysis. We establish analogies between training hyperparameters (e.g., learning rate, weight decay) and thermodynamic variables such as temperature, pressure, and volume. Starting with a simplified isotropic noise model, we uncover a close correspondence between SGD dynamics and ideal gas behavior, validated through theory and simulation. Extending to training of neural networks, we show that key predictions of the framework, including the behavior of stationary entropy, align closely with experimental observations. This framework provides a principled foundation for interpreting training dynamics and may guide future work on hyperparameter tuning and the design of learning rate schedulers.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 10 Nov 2025.
Builds on
50 of 80 references.
References · 50
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016Layer Normalizationcited by 5,669
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
- 2015Deep Learning and the Information Bottleneck Principlecited by 802
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2023Symbolic Discovery of Optimization Algorithmscited by 434
- 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
- 2018How Does Batch Normalization Help Optimization?cited by 357
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2017Three Factors Influencing Minima in SGDcited by 273
38 more
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2018Understanding Batch Normalizationcited by 159
- 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
- 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
- 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
- 2017L2 Regularization versus Batch and Weight Normalizationcited by 124
- 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
- 2018Three Mechanisms of Weight Decay Regularizationcited by 113
- 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
- 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
- 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
- 2018Theoretical Analysis of Auto Rate-Tuning by Batch Normalizationcited by 65
- 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
- 2018Norm matters: efficient and accurate normalization schemes in deep networkscited by 62
- 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
- 2024Normalization and effective learning rates in reinforcement learningcited by 58
- 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
- 2023Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networkscited by 53
- 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
- 2017Riemannian approach to batch normalizationcited by 35
- 2018Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimizationcited by 27
- 2021Learning through atypical "phase transitions" in overparameterized neural networkscited by 21
- 2018Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learningcited by 20
- 2021On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decaycited by 20
- 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
- 2022Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimescited by 19
- 2023Constructing Custom Thermodynamics Using Deep Learningcited by 17
- 2023On the different regimes of Stochastic Gradient Descentcited by 12
- 2025Neural Thermodynamic Laws for Large Language Model Trainingcited by 10
- 2022Linear Classification of Neural Manifolds with Correlated Variabilitycited by 9
- 2025Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learningcited by 9
- 2017Exploring the Function Space of Deep-Learning Machinescited by 7
- 2020A Free-Energy Principle for Representation Learningcited by 7
- 2024Where Do Large Learning Rates Lead Us?cited by 6
- 2023On the Weight Dynamics of Deep Normalized Networkscited by 4
- 2025SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Trainingcited by 4
- 2024How to explain grokkingcited by 1
Built on by
Not cited yet.