mlsanityData to 14 Oct 2026

Criticality versus uniformity in deep neural networks

arXiv 2304.04784 ↗First posted 10 Apr 2023cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency. Our results suggest that initialization along the edge of chaos is a necessary but not sufficient condition for optimal trainability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

15 of 21 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2016On the Expressive Power of Deep Neural Networkscited by 306
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  6. 2016Deep Information Propagationcited by 195
  7. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  8. 2021The Principles of Deep Learning Theorycited by 173
  9. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  10. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  11. 2020Measuring the Algorithmic Efficiency of Neural Networkscited by 52
  12. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 45
3 more
  1. 2021The edge of chaos: quantum field theory and deep neural networkscited by 24
  2. 2021Towards quantifying information flows: relative entropy in deep neural networks and the renormalization groupcited by 14
  3. 2018On the Selection of Initialization and Activation Function for Deep Neural Networkscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  2. 2023Field theory for optimal signal propagation in ResNetscited by 6
  3. 2026Deep neural networks as lattice gauge theoriescited by 0