mlsanityData to 7 Oct 2026

Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case

arXiv 2201.04543 ↗First posted 12 Jan 2022stat.ML cs.LG math.PRcited by 4

Not on the map. Why?

Abstract (arXiv)

The paper deals with the distribution of singular values of the input-output Jacobian of deep untrained neural networks in the limit of their infinite width. The Jacobian is the product of random matrices where the independent rectangular weight matrices alternate with diagonal matrices whose entries depend on the corresponding column of the nearest neighbor weight matrix. The problem was considered in citePe-Co:18 for the Gaussian weights and biases and also for the weights that are Haar distributed orthogonal matrices and Gaussian biases. Basing on a free probability argument, it was claimed that in these cases the singular value distribution of the Jacobian in the limit of infinite width (matrix size) coincides with that of the analog of the Jacobian with special random but weight independent diagonal matrices, the case well known in random matrix theory. The claim was rigorously proved in citePa-Sl:21 for a quite general class of weights and biases with i.i.d. (including Gaussian) entries by using a version of the techniques of random matrix theory. In this paper we use another version of the techniques to justify the claim for random Haar distributed weight matrices and Gaussian biases.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 15 of its 36 references that are in the map. First posted on arXiv 12 Jan 2022.

Builds on

15 of 36 references.

References · 15

  1. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  2. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  3. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  4. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  5. 2016Deep Information Propagationcited by 194
  6. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  7. 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
  8. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  9. 2015Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?cited by 40
  10. 2020Deep Randomized Neural Networkscited by 33
  11. 2017Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behaviorcited by 24
  12. 2018Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functioncited by 22
3 more
  1. 2020On Random Matrices Arising in Deep Neural Networks. Gaussian Casecited by 14
  2. 2020On Random Matrices Arising in Deep Neural Networks: General I.I.D. Casecited by 10
  3. 2018Spectrum concentration in deep residual learning: a free probability approachcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Towards Quantifying the Hessian Structure of Neural Networkscited by 13
  2. 2024Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixturescited by 6
  3. 2025Global law of conjugate kernel random matrices with heavy-tailed weightscited by 3
  4. 2025Concentration of measure for non-linear random matrices with applications to neural networks and non-commutative polynomialscited by 0