# Products of Many Large Random Matrices and Gradients in Deep Neural Networks

arXiv: https://arxiv.org/abs/1812.05994 · HTML: https://mlsanity.com/paper/1812.05994/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2018.

## Built on by (most cited first, 30 of 49)

- [Don't be lazy: CompleteP enables compute-efficient deep transformers](https://mlsanity.com/paper/2505.01618/) (2025)
- [Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit](https://mlsanity.com/paper/2309.16620/) (2023)
- [The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit](https://mlsanity.com/paper/2306.17759/) (2023)
- [Random Neural Networks in the Infinite Width Limit as Gaussian Processes](https://mlsanity.com/paper/2107.01562/) (2021)
- [Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks](https://mlsanity.com/paper/2304.03408/) (2023)
- [The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization](https://mlsanity.com/paper/2206.02768/) (2022)
- [The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization](https://mlsanity.com/paper/2106.04013/) (2021)
- [Spectral Evolution and Invariance in Linear-width Neural Networks](https://mlsanity.com/paper/2211.06506/) (2022)
- [Precise characterization of the prior predictive distribution of deep ReLU networks](https://mlsanity.com/paper/2106.06615/) (2021)
- [Super Consistency of Neural Network Landscapes and Learning Rate Transfer](https://mlsanity.com/paper/2402.17457/) (2024)
- [Bayesian Interpolation with Deep Linear Networks](https://mlsanity.com/paper/2212.14457/) (2022)
- [Width and Depth Limits Commute in Residual Networks](https://mlsanity.com/paper/2302.00453/) (2023)
- [Infinite-width limit of deep linear neural networks](https://mlsanity.com/paper/2211.16980/) (2022)
- [Meta-Principled Family of Hyperparameter Scaling Strategies](https://mlsanity.com/paper/2210.04909/) (2022)
- [Deep ReLU Networks Preserve Expected Length](https://mlsanity.com/paper/2102.10492/) (2021)
- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [Effects of Data Geometry in Early Deep Learning](https://mlsanity.com/paper/2301.00008/) (2022)
- [Maximal Initial Learning Rates in Deep ReLU Networks](https://mlsanity.com/paper/2212.07295/) (2022)
- [The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networks](https://mlsanity.com/paper/2311.18718/) (2023)
- [Injectivity of ReLU networks: perspectives from statistical physics](https://mlsanity.com/paper/2302.14112/) (2023)
- [Models of Heavy-Tailed Mechanistic Universality](https://mlsanity.com/paper/2506.03470/) (2025)
- [Deep Linear Networks for Matrix Completion -- An Infinite Depth Limit](https://mlsanity.com/paper/2210.12497/) (2022)
- [On skip connections and normalisation layers in deep optimisation](https://mlsanity.com/paper/2210.05371/) (2022)
- [Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case](https://mlsanity.com/paper/2103.13466/) (2021)
- [Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians](https://mlsanity.com/paper/2312.17306/) (2023)
- [Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks](https://mlsanity.com/paper/2310.12079/) (2023)
- [ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data](https://mlsanity.com/paper/2406.11666/) (2024)
- [Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks](https://mlsanity.com/paper/2106.03763/) (2021)
- [Free Probability for predicting the performance of feed-forward fully connected neural networks](https://mlsanity.com/paper/2111.00841/) (2021)
- [Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activations](https://mlsanity.com/paper/2306.11668/) (2023)

## Abstract (arXiv)

We study products of random matrices in the regime where the number of terms and the size of the matrices simultaneously tend to infinity. Our main theorem is that the logarithm of the ℓ2 norm of such a product applied to any fixed vector is asymptotically Gaussian. The fluctuations we find can be thought of as a finite temperature correction to the limit in which first the size and then the number of matrices tend to infinity. Depending on the scaling limit considered, the mean and variance of the limiting Gaussian depend only on either the first two or the first four moments of the measure from which matrix entries are drawn. We also obtain explicit error bounds on the moments of the norm and the Kolmogorov-Smirnov distance to a Gaussian. Finally, we apply our result to obtain precise information about the stability of gradients in randomly initialized deep neural networks with ReLU activations. This provides a quantitative measure of the extent to which the exploding and vanishing gradient problem occurs in a fully connected neural network with ReLU activations and a given architecture.