# Beyond Random Matrix Theory for Deep Networks

arXiv: https://arxiv.org/abs/2006.07721 · HTML: https://mlsanity.com/paper/2006.07721/

## Key facts

- Cited by 11 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 13 Jun 2020.

## Built on by (most cited first, 11 of 11)

- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [Hessian Eigenspectra of More Realistic Nonlinear Models](https://mlsanity.com/paper/2103.01519/) (2021)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Unifying Low Dimensional Spectra in Deep Learning](https://mlsanity.com/paper/2404.06106/) (2024)
- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [Appearance of Random Matrix Theory in Deep Learning](https://mlsanity.com/paper/2102.06740/) (2021)
- [A spin-glass model for the loss surfaces of generative adversarial networks](https://mlsanity.com/paper/2101.02524/) (2021)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)
- [Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization](https://mlsanity.com/paper/2606.30226/) (2026)
- [Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking](https://mlsanity.com/paper/2610.09919/) (2026)

## Abstract (arXiv)

We investigate whether the Wigner semi-circle and Marcenko-Pastur distributions, often used for deep neural network theoretical analysis, match empirically observed spectral densities. We find that even allowing for outliers, the observed spectral shapes strongly deviate from such theoretical predictions. This raises major questions about the usefulness of these models in deep learning. We further show that theoretical results, such as the layered nature of critical points, are strongly dependent on the use of the exact form of these limiting spectral densities. We consider two new classes of matrix ensembles; random Wigner/Wishart ensemble products and percolated Wigner/Wishart ensembles, both of which better match observed spectra. They also give large discrete spectral peaks at the origin, providing a theoretical explanation for the observation that various optima can be connected by one dimensional of low loss values. We further show that, in the case of a random matrix product, the weight of the discrete spectral component at 0 depends on the ratio of the dimensions of the weight matrices.