# Lecture Notes on Statistical Physics and Neural Networks

arXiv: https://arxiv.org/abs/2605.06394 · HTML: https://mlsanity.com/paper/2605.06394/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 48 references that are in the map.
- First posted on arXiv 7 May 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Formal Algorithms for Transformers](https://mlsanity.com/paper/2207.09238/) (2022)
- [Neural Networks and Quantum Field Theory](https://mlsanity.com/paper/2008.08601/) (2020)
- [An exact mapping between the Variational Renormalization Group and Deep Learning](https://mlsanity.com/paper/1410.3831/) (2014)
- [Scale-invariant Feature Extraction of Neural Network and Renormalization Group Flow](https://mlsanity.com/paper/1801.07172/) (2018)
- [Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group](https://mlsanity.com/paper/2107.06898/) (2021)
- [Bayesian Renormalization](https://mlsanity.com/paper/2305.10491/) (2023)
- [The Inverse of Exact Renormalization Group Flows as Statistical Inference](https://mlsanity.com/paper/2212.11379/) (2022)
- [Phase Transitions in Large Language Models and the O(N) Model](https://mlsanity.com/paper/2501.16241/) (2025)
- [NCoder -- A Quantum Field Theory approach to encoding data](https://mlsanity.com/paper/2402.00944/) (2024)
- [Is Deep Learning a Renormalization Group Flow?](https://mlsanity.com/paper/1906.05212/) (2019)

## Abstract (arXiv)

These lecture notes introduce some topics of classical statistical physics, particularly those that are relevant for neural networks and deep learning. Statistical physics is treated as a branch of probability theory or statistics, with the goal of making concepts such as phase transitions and the renormalization group accessible to readers without prior knowledge of physics. We introduce the Boltzmann-Gibbs distribution and the thermodynamic potentials on a finite configuration space, notably for Ising spins and spin-glass models on a lattice, and then define phase transitions as discontinuities that arise in the limit that the number of lattice points goes to infinity. We further introduce Hopfield networks and Boltzmann machines, which are governed by the same energy function as spin-glass models, and discuss the learning algorithm for restricted Boltzmann machines. In this algorithm hidden neurons are integrated out as in the renormalization group. Finally, modern deep learning is introduced, whose early developments were in part motivated by restricted Boltzmann machines in that they carry many layers of hidden neurons. A description of large language models is given.