# Statistical Mechanics and Artificial Neural Networks: Principles, Models, and Applications

arXiv: https://arxiv.org/abs/2405.10957 · HTML: https://mlsanity.com/paper/2405.10957/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 141 references that are in the map.
- First posted on arXiv 5 Apr 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [PyHessian: Neural Networks Through the Lens of the Hessian](https://mlsanity.com/paper/1912.07145/) (2019)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [On Empirical Comparisons of Optimizers for Deep Learning](https://mlsanity.com/paper/1910.05446/) (2019)
- [Minimum Width for Universal Approximation](https://mlsanity.com/paper/2006.08859/) (2020)
- [Visualizing and Understanding the Effectiveness of BERT](https://mlsanity.com/paper/1908.05620/) (2019)
- [Data-driven discovery of Koopman eigenfunctions for control](https://mlsanity.com/paper/1707.01146/) (2017)
- [Neural ODE control for classification, approximation and transport](https://mlsanity.com/paper/2104.05278/) (2021)
- [The gap between theory and practice in function approximation with deep neural networks](https://mlsanity.com/paper/2001.07523/) (2020)
- [Optimal control of PDEs using physics-informed neural networks](https://mlsanity.com/paper/2111.09880/) (2021)
- [Hessian Eigenspectra of More Realistic Nonlinear Models](https://mlsanity.com/paper/2103.01519/) (2021)
- [Implicit energy regularization of neural ordinary-differential-equation control](https://mlsanity.com/paper/2103.06525/) (2021)
- [Neural Ordinary Differential Equation Control of Dynamics on Graphs](https://mlsanity.com/paper/2006.09773/) (2020)
- [Interpretable Polynomial Neural Ordinary Differential Equations](https://mlsanity.com/paper/2208.05072/) (2022)
- [Visualizing high-dimensional loss landscapes with Hessian directions](https://mlsanity.com/paper/2208.13219/) (2022)
- [Exploring the Geometry and Topology of Neural Network Loss Landscapes](https://mlsanity.com/paper/2102.00485/) (2021)
- [Learning the Ising Model with Generative Neural Networks](https://mlsanity.com/paper/2001.05361/) (2020)
- [Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks](https://mlsanity.com/paper/2201.00032/) (2021)
- [Spectrally Adapted Physics-Informed Neural Networks for Solving Unbounded Domain Problems](https://mlsanity.com/paper/2202.02710/) (2022)
- [Near-optimal control of dynamical systems with neural ordinary differential equations](https://mlsanity.com/paper/2206.11120/) (2022)

## Abstract (arXiv)

The field of neuroscience and the development of artificial neural networks (ANNs) have mutually influenced each other, drawing from and contributing to many concepts initially developed in statistical mechanics. Notably, Hopfield networks and Boltzmann machines are versions of the Ising model, a model extensively studied in statistical mechanics for over a century. In the first part of this chapter, we provide an overview of the principles, models, and applications of ANNs, highlighting their connections to statistical mechanics and statistical learning theory. Artificial neural networks can be seen as high-dimensional mathematical functions, and understanding the geometric properties of their loss landscapes (i.e., the high-dimensional space on which one wishes to find extrema or saddles) can provide valuable insights into their optimization behavior, generalization abilities, and overall performance. Visualizing these functions can help us design better optimization methods and improve their generalization abilities. Thus, the second part of this chapter focuses on quantifying geometric properties and visualizing loss functions associated with deep ANNs.