mlsanityData to 7 Oct 2026

Deep Information Propagation

arXiv 1611.01232 ↗First posted 4 Nov 2016stat.ML cs.LGcited by 194

Not on the map. Why?

Abstract (arXiv)

We study the behavior of untrained neural networks whose weights and biases are randomly distributed using mean field theory. We show the existence of depth scales that naturally limit the maximum depth of signal propagation through these random networks. Our main practical result is to show that random networks may be trained precisely when information can travel through them. Thus, the depth scales that we identify provide bounds on how deep a network may be trained for a specific choice of hyperparameters. As a corollary to this, we argue that in networks at the edge of chaos, one of these depth scales diverges. Thus arbitrarily deep networks may be trained only sufficiently close to criticality. We show that the presence of dropout destroys the order-to-chaos critical point and therefore strongly limits the maximum trainable depth for random networks. Finally, we develop a mean field theory for backpropagation and we show that the ordered and chaotic phases correspond to regions of vanishing and exploding gradient respectively.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 35; 2022: 35; 2023: 39; 2024: 20; 2025: 32; 2026: 33.

194 papers.

Sort:

By area:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2021Explaining Neural Scaling Lawscited by 353
  3. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  4. 2023Understanding plasticity in neural networkscited by 170
  5. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  6. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  7. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 106
  8. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
  9. 2021Improved architectures and training algorithms for deep operator networkscited by 85
  10. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  11. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  12. 2022Fast Finite Width Neural Tangent Kernelcited by 62
38 more
  1. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  2. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  3. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  4. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  5. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  6. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  7. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  8. 2023Simplifying Transformer Blockscited by 37
  9. 2022Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth and Initializationcited by 30
  10. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  11. 2021Batch Normalization Orthogonalizes Representations in Deep Random Networkscited by 28
  12. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  13. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  14. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  15. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  16. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  17. 2021Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learningcited by 24
  18. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  19. 2021The edge of chaos: quantum field theory and deep neural networkscited by 23
  20. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  21. 2022Characterizing the Spectrum of the NTK via a Power Series Expansioncited by 22
  22. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  23. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  24. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  25. 2023Effective Theory of Transformers at Initializationcited by 21
  26. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  27. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  28. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  29. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  30. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  31. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 18
  32. 2021Eigenspace Restructuring: a Principle of Space and Frequency in Neural Networkscited by 17
  33. 2025Beyond Zero Initialization: Investigating the Impact of Non-Zero Initialization on LoRA Fine-Tuning Dynamicscited by 16
  34. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  35. 2021Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspectivecited by 14
  36. 2021Convergence of Deep ReLU Networkscited by 14
  37. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  38. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14

50 most cited of 194 All 194 (JSON)