mlsanityData to 14 Oct 2026

Exact Phase Transitions in Deep Learning

arXiv 2205.12510 ↗First posted 25 May 2022cs.LG cond-mat.dis-nn physics.app-phcited by 6

Not on the map. Why?

Abstract (arXiv)

This work reports deep-learning-unique first-order and second-order phase transitions, whose phenomenology closely follows that in statistical physics. In particular, we prove that the competition between prediction error and model complexity in the training loss leads to the second-order phase transition for nets with one hidden layer and the first-order phase transition for nets with more than one hidden layer. The proposed theory is directly relevant to the optimization of neural networks and points to an origin of the posterior collapse problem in Bayesian deep learning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 13 of its 26 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

13 of 26 references.

References · 13

  1. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  2. 2016Deep Learning without Poor Local Minimacited by 234
  3. 2019Two models of double descent for weak featurescited by 195
  4. 2019Diagnosing and Enhancing VAE Modelscited by 193
  5. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 121
  6. 2016Identity Matters in Deep Learningcited by 109
  7. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  8. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 70
  9. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
  10. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 50
  11. 2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 25
  12. 2022Exact Solutions of a Deep Linear Networkcited by 24
1 more
  1. 2019On Dropout and Nuclear Norm Regularizationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 0; 2026: 1.

6 papers.

  1. 2023Symmetry Induces Structure and Constraint of Learningcited by 25
  2. 2022What shapes the loss landscape of self-supervised learning?cited by 19
  3. 2023An effective theory of collective deep learningcited by 7
  4. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  5. 2023Scaling and Resizing Symmetry in Feedforward Networkscited by 0
  6. 2026Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weightscited by 0