# Exact Phase Transitions in Deep Learning

arXiv: https://arxiv.org/abs/2205.12510 · HTML: https://mlsanity.com/paper/2205.12510/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 13 of 13)

- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Diagnosing and Enhancing VAE Models](https://mlsanity.com/paper/1903.05789/) (2019)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Identity Matters in Deep Learning](https://mlsanity.com/paper/1611.04231/) (2016)
- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Deep linear neural networks with arbitrary loss: All local minima are global](https://mlsanity.com/paper/1712.01473/) (2017)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior](https://mlsanity.com/paper/1710.09553/) (2017)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [On Dropout and Nuclear Norm Regularization](https://mlsanity.com/paper/1905.11887/) (2019)

## Built on by (most cited first, 6 of 6)

- [Symmetry Induces Structure and Constraint of Learning](https://mlsanity.com/paper/2309.16932/) (2023)
- [What shapes the loss landscape of self-supervised learning?](https://mlsanity.com/paper/2210.00638/) (2022)
- [An effective theory of collective deep learning](https://mlsanity.com/paper/2310.12802/) (2023)
- [Phase Transitions in the Output Distribution of Large Language Models](https://mlsanity.com/paper/2405.17088/) (2024)
- [Scaling and Resizing Symmetry in Feedforward Networks](https://mlsanity.com/paper/2306.15015/) (2023)
- [Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weights](https://mlsanity.com/paper/2607.27482/) (2026)

## Abstract (arXiv)

This work reports deep-learning-unique first-order and second-order phase transitions, whose phenomenology closely follows that in statistical physics. In particular, we prove that the competition between prediction error and model complexity in the training loss leads to the second-order phase transition for nets with one hidden layer and the first-order phase transition for nets with more than one hidden layer. The proposed theory is directly relevant to the optimization of neural networks and points to an origin of the posterior collapse problem in Bayesian deep learning.