# Appearance of Random Matrix Theory in Deep Learning

arXiv: https://arxiv.org/abs/2102.06740 · HTML: https://mlsanity.com/paper/2102.06740/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 80 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [An Investigation into Neural Net Optimization via Hessian Eigenvalue Density](https://mlsanity.com/paper/1901.10159/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond](https://mlsanity.com/paper/1611.07476/) (2016)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Generalisation error in learning with random features and the hidden manifold model](https://mlsanity.com/paper/2002.09339/) (2020)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [The Gaussian equivalence of generative models for learning with shallow neural networks](https://mlsanity.com/paper/2006.14709/) (2020)
- [The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization](https://mlsanity.com/paper/2008.06786/) (2020)
- [Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians](https://mlsanity.com/paper/1901.08244/) (2019)
- [Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems](https://mlsanity.com/paper/1905.11675/) (2019)
- [Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Model](https://mlsanity.com/paper/1907.08226/) (2019)
- [Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions](https://mlsanity.com/paper/1804.02686/) (2018)
- [Mean-field inference methods for neural networks](https://mlsanity.com/paper/1911.00890/) (2019)
- [Landscape Complexity for the Empirical Risk of Generalized Linear Models](https://mlsanity.com/paper/1912.02143/) (2019)
- [Beyond Random Matrix Theory for Deep Networks](https://mlsanity.com/paper/2006.07721/) (2020)
- [Explorations on high dimensional landscapes](https://mlsanity.com/paper/1412.6615/) (2014)
- [The Loss Surfaces of Neural Networks with General Activation Functions](https://mlsanity.com/paper/2004.03959/) (2020)
- [A spin-glass model for the loss surfaces of generative adversarial networks](https://mlsanity.com/paper/2101.02524/) (2021)
- [On the energy landscape of deep networks](https://mlsanity.com/paper/1511.06485/) (2015)
- [Topology trivialization and large deviations for the minimum in the simplest random optimization](https://mlsanity.com/paper/1304.0024/) (2013)
- [Deep Curvature Suite](https://mlsanity.com/paper/1912.09656/) (2019)

## Built on by (most cited first, 7 of 7)

- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learning](https://mlsanity.com/paper/2401.03137/) (2024)
- [Stochastic weight matrix dynamics during learning and Dyson Brownian motion](https://mlsanity.com/paper/2407.16427/) (2024)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)
- [Random Matrix Theory for Stochastic Gradient Descent](https://mlsanity.com/paper/2412.20496/) (2024)
- [Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generation](https://mlsanity.com/paper/2510.17120/) (2025)
- [Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory](https://mlsanity.com/paper/2512.12911/) (2025)

## Abstract (arXiv)

We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets. These results shed new light on the applicability of Random Matrix Theory to modelling neural networks and suggest a previously unrecognised role for it in the study of loss surfaces in deep learning. Inspired by these observations, we propose a novel model for the true loss surfaces of neural networks, consistent with our observations, which allows for Hessian spectral densities with rank degeneracy and outliers, extensively observed in practice, and predicts a growing independence of loss gradients as a function of distance in weight-space. We further investigate the importance of the true loss surface in neural networks and find, in contrast to previous work, that the exponential hardness of locating the global minimum has practical consequences for achieving state of the art performance.