mlsanityData to 7 Oct 2026

Deep Neural Nets as Hamiltonians

arXiv 2503.23982 ↗First posted 31 Mar 2025cond-mat.dis-nn cond-mat.stat-mech cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural networks are complex functions of both their inputs and parameters. Much prior work in deep learning theory analyzes the distribution of network outputs at a fixed a set of inputs (e.g. a training dataset) over random initializations of the network parameters. The purpose of this article is to consider the opposite situation: we view a randomly initialized Multi-Layer Perceptron (MLP) as a Hamiltonian over its inputs. For typical realizations of the network parameters, we study the properties of the energy landscape induced by this Hamiltonian, focusing on the structure of near-global minimum in the limit of infinite width. Specifically, we use the replica trick to perform an exact analytic calculation giving the entropy (log volume of space) at a given energy. We further derive saddle point equations that describe the overlaps between inputs sampled iid from the Gibbs distribution induced by the random MLP. For linear activations we solve these saddle point equations exactly. But we also solve them numerically for a variety of depths and activation functions, including tanh, sin, ReLU, and shaped non-linearities. We find even at infinite width a rich range of behaviors. For some non-linearities, such as sin, for instance, we find that the landscapes of random MLPs exhibit full replica symmetry breaking, while shallow tanh and ReLU networks or deep shaped MLPs are instead replica symmetric.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

33 of 56 references.

References · 33

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2014How transferable are features in deep neural networks?cited by 1,216
  3. 2017Deep Neural Networks as Gaussian Processescited by 617
  4. 2015Understanding Neural Networks Through Deep Visualizationcited by 332
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2014The Loss Surfaces of Multilayer Networkscited by 278
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  8. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  9. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  10. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  11. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  12. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 94
21 more
  1. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  2. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  3. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  4. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  5. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  6. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  7. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  8. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  9. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  10. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  11. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  12. 2021A single gradient step finds adversarial examples on random two-layers neural networkscited by 26
  13. 2021Adversarial Examples in Multi-Layer Random ReLU Networkscited by 26
  14. 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
  15. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  16. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  17. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  18. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  19. 2024High-dimensional learning of narrow neural networkscited by 12
  20. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  21. 2024Exploring Loss Landscapes through the Lens of Spin Glass Theorycited by 4

Built on by

Not cited yet.