# On Random Matrices Arising in Deep Neural Networks. Gaussian Case

arXiv: https://arxiv.org/abs/2001.06188 · HTML: https://mlsanity.com/paper/2001.06188/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2020.

## Built on by (most cited first, 14 of 14)

- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Overparameterized Linear Regression under Adversarial Attacks](https://mlsanity.com/paper/2204.06274/) (2022)
- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [Concentration of Random Feature Matrices in High-Dimensions](https://mlsanity.com/paper/2204.06935/) (2022)
- [Random matrices in service of ML footprint: ternary random features with no performance loss](https://mlsanity.com/paper/2110.01899/) (2021)
- [Conditioning of Random Feature Matrices: Double Descent and Generalization Error](https://mlsanity.com/paper/2110.11477/) (2021)
- [Asymptotic Freeness of Layerwise Jacobians Caused by Invariance of Multilayer Perceptron: The Haar Orthogonal Case](https://mlsanity.com/paper/2103.13466/) (2021)
- [Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case](https://mlsanity.com/paper/2201.04543/) (2022)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)
- [Free Probability for predicting the performance of feed-forward fully connected neural networks](https://mlsanity.com/paper/2111.00841/) (2021)
- ["Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach](https://mlsanity.com/paper/2403.00258/) (2024)
- [Pruning Deep Neural Networks via the Marchenko--Pastur Distribution](https://mlsanity.com/paper/2606.02608/) (2026)
- [Randomly initialized autoencoders: fixed points and edge-of-chaos](https://mlsanity.com/paper/2608.14638/) (2026)
- [On the Principles Behind Neural Network Optimizers](https://mlsanity.com/paper/2608.16760/) (2026)

## Abstract (arXiv)

The paper deals with distribution of singular values of product of random matrices arising in the analysis of deep neural networks. The matrices resemble the product analogs of the sample covariance matrices, however, an important difference is that the population covariance matrices, which are assumed to be non-random in the standard setting of statistics and random matrix theory, are now random, moreover, are certain functions of random data matrices. The problem has been considered in recent work [21] by using the techniques of free probability theory. Since, however, free probability theory deals with population matrices which are independent of the data matrices, its applicability in this case requires an additional justification. We present this justification by using a version of the standard techniques of random matrix theory under the assumption that the entries of data matrices are independent Gaussian random variables. In the subsequent paper [18] we extend our results to the case where the entries of data matrices are just independent identically distributed random variables with several finite moments. This, in particular, extends the property of the so-called macroscopic universality on the considered random matrices.