mlsanityData to 7 Oct 2026

Criticality versus uniformity in deep neural networks

arXiv 2304.04784 ↗First posted 10 Apr 2023cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep feedforward networks initialized along the edge of chaos exhibit exponentially superior training ability as quantified by maximum trainable depth. In this work, we explore the effect of saturation of the tanh activation function along the edge of chaos. In particular, we determine the line of uniformity in phase space along which the post-activation distribution has maximum entropy. This line intersects the edge of chaos, and indicates the regime beyond which saturation of the activation function begins to impede training efficiency. Our results suggest that initialization along the edge of chaos is a necessary but not sufficient condition for optimal trainability.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

15 of 21 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2016On the Expressive Power of Deep Neural Networkscited by 302
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  6. 2016Deep Information Propagationcited by 194
  7. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  8. 2021The Principles of Deep Learning Theorycited by 169
  9. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  10. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  11. 2020Measuring the Algorithmic Efficiency of Neural Networkscited by 51
  12. 2018Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networkscited by 44
3 more
  1. 2021The edge of chaos: quantum field theory and deep neural networkscited by 23
  2. 2021Towards quantifying information flows: relative entropy in deep neural networks and the renormalization groupcited by 12
  3. 2018On the Selection of Initialization and Activation Function for Deep Neural Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  2. 2023Field theory for optimal signal propagation in ResNetscited by 6
  3. 2026Deep neural networks as lattice gauge theoriescited by 0