# Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchies

arXiv: https://arxiv.org/abs/2204.01058 · HTML: https://mlsanity.com/paper/2204.01058/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2022.

## Built on by (most cited first, 24 of 24)

- [Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks](https://mlsanity.com/paper/2304.03408/) (2023)
- [Bayesian Interpolation with Deep Linear Networks](https://mlsanity.com/paper/2212.14457/) (2022)
- [Quantitative CLTs in Deep Neural Networks](https://mlsanity.com/paper/2307.06092/) (2023)
- [Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks](https://mlsanity.com/paper/2307.11807/) (2023)
- [Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning](https://mlsanity.com/paper/2510.02091/) (2025)
- [Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers](https://mlsanity.com/paper/2406.03260/) (2024)
- [From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning](https://mlsanity.com/paper/2502.03210/) (2025)
- [Bayesian Inference with Deep Weakly Nonlinear Networks](https://mlsanity.com/paper/2405.16630/) (2024)
- [Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks](https://mlsanity.com/paper/2307.02284/) (2023)
- [Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization](https://mlsanity.com/paper/2302.09712/) (2023)
- [Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations](https://mlsanity.com/paper/2306.11668/) (2023)
- [Proportional infinite-width infinite-depth limit for deep linear neural networks](https://mlsanity.com/paper/2411.15267/) (2024)
- [Online Federation For Mixtures of Proprietary Agents with Black-Box Encoders](https://mlsanity.com/paper/2505.00216/) (2025)
- [Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime](https://mlsanity.com/paper/2605.29684/) (2026)
- [Bayesian Inference with Shaped Deep Non-linear MLPs](https://mlsanity.com/paper/2605.30860/) (2026)
- [Network Degeneracy as an Indicator of Training Performance: Comparing Finite and Infinite Width Angle Predictions](https://mlsanity.com/paper/2306.01513/) (2023)
- [Deep Neural Nets as Hamiltonians](https://mlsanity.com/paper/2503.23982/) (2025)
- [Large deviation principles for convolutional Bayesian neural networks](https://mlsanity.com/paper/2603.06023/) (2026)
- [Loop Corrections in Random Feature Models: Training Error and Generalization Gap](https://mlsanity.com/paper/2604.12827/) (2026)
- [Collective Kernel EFT for Pre-activation ResNets](https://mlsanity.com/paper/2604.15742/) (2026)
- [How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences](https://mlsanity.com/paper/2605.05113/) (2026)
- [Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputs](https://mlsanity.com/paper/2605.24072/) (2026)
- [High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercube](https://mlsanity.com/paper/2608.26526/) (2026)
- [Critical initialization destabilizes higher input derivatives in wide scalar-input networks](https://mlsanity.com/paper/2609.09244/) (2026)

## Abstract (arXiv)

This article considers fully connected neural networks with Gaussian random weights and biases as well as L hidden layers, each of width proportional to a large parameter n. For polynomially bounded non-linearities we give sharp estimates in powers of 1/n for the joint cumulants of the network output and its derivatives. Moreover, we show that network cumulants form a perturbatively solvable hierarchy in powers of 1/n in that k-th order cumulants in one layer have recursions that depend to leading order in 1/n only on j-th order cumulants at the previous layer with j≤ k. By solving a variety of such recursions, however, we find that the depth-to-width ratio L/n plays the role of an effective network depth, controlling both the scale of fluctuations at individual neurons and the size of inter-neuron correlations. Thus, while the cumulant recursions we derive form a hierarchy in powers of 1/n, contributions of order 1/nk often grow like Lk and are hence non-negligible at positive L/n. We use this to study a somewhat simplified version of the exploding and vanishing gradient problem, proving that this particular variant occurs if and only if L/n is large. Several key ideas in this article were first developed at a physics level of rigor in a recent monograph of Daniel A. Roberts, Sho Yaida, and the author. This article not only makes these ideas mathematically precise but also significantly extends them, opening the way to obtaining corrections to all orders in 1/n.