mlsanityData to 7 Oct 2026

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks

arXiv 2307.02284 ↗First posted 5 Jul 2023stat.ML cond-mat.dis-nn cond-mat.stat-mechcited by 4

Not on the map. Why?

Abstract (arXiv)

We demonstrate that conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics, also known as the edge of chaos, exhibit universal scaling laws of absorbing phase transitions in non-equilibrium statistical mechanics. We exploit the fully deterministic nature of the propagation dynamics to elucidate an analogy between a signal collapse in the neural networks and an absorbing state (a state that the system can enter but cannot escape from). Our numerical results indicate that the multilayer perceptrons and the convolutional neural networks belong to the mean-field and the directed percolation universality classes, respectively. Also, the finite-size scaling is successfully applied, suggesting a potential connection to the depth-width trade-off in deep learning. Furthermore, our analysis of the training dynamics under the gradient descent reveals that hyperparameter tuning to the phase boundary is necessary but insufficient for achieving optimal generalization in deep networks. Remarkably, nonuniversal metric factors associated with the scaling laws are shown to play a significant role in concretizing the above observations. These findings highlight the usefulness of the notion of criticality for analyzing the behavior of artificial deep neural networks and offer new insights toward a unified understanding of the essential relationship between criticality and intelligence.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 29 of its 80 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

29 of 80 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2017Deep Neural Networks as Gaussian Processescited by 617
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  9. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  10. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  11. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  12. 2016Deep Information Propagationcited by 194
17 more
  1. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  2. 2021The Principles of Deep Learning Theorycited by 169
  3. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 153
  4. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  5. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  6. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  7. 2019Neural Machine Translation: A Review and Surveycited by 99
  8. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  9. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  10. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  11. 2020On the infinite width limit of neural networks with a standard parameterizationcited by 38
  12. 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
  13. 2021The edge of chaos: quantum field theory and deep neural networkscited by 23
  14. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  15. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  16. 2023Suppression of chaos in a partially driven recurrent neural networkcited by 3
  17. 2021A devil's advocate view on 'self-organized' brain criticalitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 21
  2. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7
  3. 2025Toward a Physics of Deep Learning and Brainscited by 1
  4. 2026Quantum Tunneling-Aware Machine Learning: Physics-Derived Noise Models for Robust Deploymentcited by 0