# On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case

arXiv: https://arxiv.org/abs/2011.11439 · HTML: https://mlsanity.com/paper/2011.11439/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2020.

## Built on by (most cited first, 10 of 10)

- [Towards Quantifying the Hessian Structure of Neural Networks](https://mlsanity.com/paper/2505.02809/) (2025)
- [Concentration of Random Feature Matrices in High-Dimensions](https://mlsanity.com/paper/2204.06935/) (2022)
- [Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures](https://mlsanity.com/paper/2402.02697/) (2024)
- [Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case](https://mlsanity.com/paper/2201.04543/) (2022)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)
- [Free Probability for predicting the performance of feed-forward fully connected neural networks](https://mlsanity.com/paper/2111.00841/) (2021)
- [Random weights of DNNs and emergence of fixed points](https://mlsanity.com/paper/2501.04182/) (2025)
- [Pruning Deep Neural Networks via the Marchenko--Pastur Distribution](https://mlsanity.com/paper/2606.02608/) (2026)
- [Randomly initialized autoencoders: fixed points and edge-of-chaos](https://mlsanity.com/paper/2608.14638/) (2026)
- [High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercube](https://mlsanity.com/paper/2608.26526/) (2026)

## Abstract (arXiv)

We study the distribution of singular values of product of random matrices pertinent to the analysis of deep neural networks. The matrices resemble the product of the sample covariance matrices, however, an important difference is that the population covariance matrices assumed to be non-random or random but independent of the random data matrix in statistics and random matrix theory are now certain functions of random data matrices (synaptic weight matrices in the deep neural network terminology). The problem has been treated in recent work [25, 13] by using the techniques of free probability theory. Since, however, free probability theory deals with population covariance matrices which are independent of the data matrices, its applicability has to be justified. The justification has been given in [22] for Gaussian data matrices with independent entries, a standard analytical model of free probability, by using a version of the techniques of random matrix theory. In this paper we use another, more streamlined, version of the techniques of random matrix theory to generalize the results of [22] to the case where the entries of the synaptic weight matrices are just independent identically distributed random variables with zero mean and finite fourth moment. This, in particular, extends the property of the so-called macroscopic universality on the considered random matrices.