mlsanityData to 7 Oct 2026

Exponential expressivity in deep neural networks through transient chaos

arXiv 1606.05340 ↗First posted 16 Jun 2016stat.ML cond-mat.dis-nn cs.LGcited by 260

Not on the map. Why?

Abstract (arXiv)

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitatively demonstrate the long conjectured idea that deep networks can disentangle highly curved manifolds in input space into flat manifolds in hidden space. Our theoretical analysis of the expressive power of deep networks broadly applies to arbitrary nonlinearities, and provides a quantitative underpinning for previously abstract notions about the geometry of deep functions.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 40; 2023: 63; 2024: 28; 2025: 42; 2026: 46.

260 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  2. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
  3. 2023Understanding plasticity in neural networkscited by 170
  4. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  5. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  6. 2023A Spectral Condition for Feature Learningcited by 101
  7. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  8. 2023Deep Directly-Trained Spiking Neural Networks for Object Detectioncited by 76
  9. 2021Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognitioncited by 71
  10. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  11. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  12. 2021Model Complexity of Deep Learning: A Surveycited by 64
38 more
  1. 2024Normalization and effective learning rates in reinforcement learningcited by 55
  2. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  3. 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 45
  4. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  5. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  6. 2025On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learningcited by 42
  7. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  8. 2023Simplifying Transformer Blockscited by 37
  9. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  10. 2021Self-learning Machines based on Hamiltonian Echo Backpropagationcited by 28
  11. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  12. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  13. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  14. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  15. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  16. 2022Image sensing with multilayer, nonlinear optical neural networkscited by 25
  17. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  18. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  19. 2021The edge of chaos: quantum field theory and deep neural networkscited by 23
  20. 2023LEMON: Lossless model expansioncited by 23
  21. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  22. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  23. 2023Generative learning for nonlinear dynamicscited by 22
  24. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  25. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  26. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  27. 2023Effective Theory of Transformers at Initializationcited by 21
  28. 2023Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaoscited by 21
  29. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  30. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  31. 2023Phase diagram of early training dynamics in deep neural networks: effect of the learning rate, depth, and widthcited by 19
  32. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  33. 2021Depth separation beyond radial functionscited by 18
  34. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  35. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  36. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  37. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  38. 2021Optimization-Based Separations for Neural Networkscited by 17

50 most cited of 260 All 260 (JSON)