# Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case

arXiv: https://arxiv.org/abs/2201.04543 · HTML: https://mlsanity.com/paper/2201.04543/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 12 Jan 2022.

## Builds on (most cited first, 15 of 15)

- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [The Emergence of Spectral Universality in Deep Networks](https://mlsanity.com/paper/1802.09979/) (2018)
- [Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks](https://mlsanity.com/paper/2001.05992/) (2020)
- [Tensor Programs III: Neural Matrix Laws](https://mlsanity.com/paper/2009.10685/) (2020)
- [Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?](https://mlsanity.com/paper/1504.08291/) (2015)
- [Deep Randomized Neural Networks](https://mlsanity.com/paper/2002.12287/) (2020)
- [Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior](https://mlsanity.com/paper/1710.09553/) (2017)
- [Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function](https://mlsanity.com/paper/1809.08848/) (2018)
- [On Random Matrices Arising in Deep Neural Networks. Gaussian Case](https://mlsanity.com/paper/2001.06188/) (2020)
- [On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case](https://mlsanity.com/paper/2011.11439/) (2020)
- [Spectrum concentration in deep residual learning: a free probability approach](https://mlsanity.com/paper/1807.11694/) (2018)

## Built on by (most cited first, 4 of 4)

- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures](https://mlsanity.com/paper/2402.02697/) (2024)
- [Global law of conjugate kernel random matrices with heavy-tailed weights](https://mlsanity.com/paper/2502.18428/) (2025)
- [Concentration of measure for non-linear random matrices with applications to neural networks and non-commutative polynomials](https://mlsanity.com/paper/2507.07625/) (2025)

## Abstract (arXiv)

The paper deals with the distribution of singular values of the input-output Jacobian of deep untrained neural networks in the limit of their infinite width. The Jacobian is the product of random matrices where the independent rectangular weight matrices alternate with diagonal matrices whose entries depend on the corresponding column of the nearest neighbor weight matrix. The problem was considered in citePe-Co:18 for the Gaussian weights and biases and also for the weights that are Haar distributed orthogonal matrices and Gaussian biases. Basing on a free probability argument, it was claimed that in these cases the singular value distribution of the Jacobian in the limit of infinite width (matrix size) coincides with that of the analog of the Jacobian with special random but weight independent diagonal matrices, the case well known in random matrix theory. The claim was rigorously proved in citePa-Sl:21 for a quite general class of weights and biases with i.i.d. (including Gaussian) entries by using a version of the techniques of random matrix theory. In this paper we use another version of the techniques to justify the claim for random Haar distributed weight matrices and Gaussian biases.