mlsanityData to 7 Oct 2026

Products of Many Large Random Matrices and Gradients in Deep Neural Networks

arXiv 1812.05994 ↗First posted 14 Dec 2018math.PR math-ph math.MPcited by 49

Not on the map. Why?

Abstract (arXiv)

We study products of random matrices in the regime where the number of terms and the size of the matrices simultaneously tend to infinity. Our main theorem is that the logarithm of the ℓ2 norm of such a product applied to any fixed vector is asymptotically Gaussian. The fluctuations we find can be thought of as a finite temperature correction to the limit in which first the size and then the number of matrices tend to infinity. Depending on the scaling limit considered, the mean and variance of the limiting Gaussian depend only on either the first two or the first four moments of the measure from which matrix entries are drawn. We also obtain explicit error bounds on the moments of the norm and the Kolmogorov-Smirnov distance to a Gaussian. Finally, we apply our result to obtain precise information about the stability of gradients in randomly initialized deep neural networks with ReLU activations. This provides a quantitative measure of the extent to which the exploding and vanishing gradient problem occurs in a fully connected neural network with ReLU activations and a given architecture.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 11; 2023: 13; 2024: 2; 2025: 7; 2026: 9.

49 papers.

Sort:

  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  2. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  3. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  4. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  5. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  6. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  7. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  8. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  9. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  10. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  11. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  12. 2023Width and Depth Limits Commute in Residual Networkscited by 21
37 more
  1. 2022Infinite-width limit of deep linear neural networkscited by 20
  2. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  3. 2021Deep ReLU Networks Preserve Expected Lengthcited by 17
  4. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  5. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  6. 2022Maximal Initial Learning Rates in Deep ReLU Networkscited by 11
  7. 2023The Feature Speed Formula: a flexible approach to scale hyper-parameters of deep neural networkscited by 11
  8. 2023Injectivity of ReLU networks: perspectives from statistical physicscited by 10
  9. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  10. 2022Deep Linear Networks for Matrix Completion -- An Infinite Depth Limitcited by 8
  11. 2022On skip connections and normalisation layers in deep optimisationcited by 7
  12. 2021Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Casecited by 5
  13. 2023Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobianscited by 5
  14. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
  15. 2024ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Datacited by 4
  16. 2021Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networkscited by 3
  17. 2021Free Probability for predicting the performance of feed-forward fully connected neural networkscited by 3
  18. 2023Principles for Initialization and Architecture Selection in Graph Neural Networks with ReLU Activationscited by 3
  19. 2025On the Stability of the Jacobian Matrix in Deep Neural Networkscited by 3
  20. 2022Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theorycited by 2
  21. 2023Upper and lower bounds for the Lipschitz constant of random neural networkscited by 2
  22. 2025Revisiting Glorot Initialization for Long-Range Linear Recurrencescited by 2
  23. 2023Expected Gradients of Maxout Networks and Consequences to Parameter Initializationcited by 1
  24. 2023Les Houches Lectures on Deep Learning at Large & Infinite Widthcited by 1
  25. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  26. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  27. 2022Dynamic mean field programmingcited by 0
  28. 2025Deep Neural Nets as Hamiltonianscited by 0
  29. 2025Free Random Projection for In-Context Reinforcement Learningcited by 0
  30. 2026How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrencescited by 0
  31. 2026Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponentcited by 0
  32. 2026Exploding and vanishing gradients in deep neural networks: the effect of residual connectionscited by 0
  33. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0
  34. 2026Randomly initialized autoencoders: fixed points and edge-of-chaoscited by 0
  35. 2026High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercubecited by 0
  36. 2026Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian Modelscited by 0
  37. 2026High-Probability Guarantees for SGD under β-Heavy-Tailed Gradient Noisecited by 0