mlsanityData to 7 Oct 2026

Appearance of Random Matrix Theory in Deep Learning

arXiv 2102.06740 ↗First posted 12 Feb 2021cs.LG math-ph math.MPcited by 7

Not on the map. Why?

Abstract (arXiv)

We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets. These results shed new light on the applicability of Random Matrix Theory to modelling neural networks and suggest a previously unrecognised role for it in the study of loss surfaces in deep learning. Inspired by these observations, we propose a novel model for the true loss surfaces of neural networks, consistent with our observations, which allows for Hessian spectral densities with rank degeneracy and outliers, extensively observed in practice, and predicts a growing independence of loss gradients as a function of distance in weight-space. We further investigate the importance of the true loss surface in neural networks and find, in contrast to previous work, that the exponential hardness of locating the global minimum has practical consequences for achieving state of the art performance.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 30 of its 80 references that are in the map. First posted on arXiv 12 Feb 2021.

Builds on

30 of 80 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2014New insights and perspectives on the natural gradient methodcited by 435
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  8. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  9. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  10. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  11. 2014The Loss Surfaces of Multilayer Networkscited by 278
  12. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 156
18 more
  1. 2015Statistical physics of inference: Thresholds and algorithmscited by 128
  2. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  3. 2019Exact Gaussian Processes on a Million Data Pointscited by 118
  4. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  5. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  6. 2019Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessianscited by 68
  7. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40
  8. 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
  9. 2018Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitionscited by 23
  10. 2019Mean-field inference methods for neural networkscited by 16
  11. 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
  12. 2020Beyond Random Matrix Theory for Deep Networkscited by 11
  13. 2014Explorations on high dimensional landscapescited by 8
  14. 2020The Loss Surfaces of Neural Networks with General Activation Functionscited by 7
  15. 2021A spin-glass model for the loss surfaces of generative adversarial networkscited by 4
  16. 2015On the energy landscape of deep networkscited by 3
  17. 2013Topology trivialization and large deviations for the minimum in the simplest random optimizationcited by 1
  18. 2019Deep Curvature Suitecited by 1

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 0; 2024: 3; 2025: 2.

7 papers.

  1. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  2. 2024SPQR: Controlling Q-ensemble Independence with Spiked Random Model for Reinforcement Learningcited by 8
  3. 2024Stochastic weight matrix dynamics during learning and Dyson Brownian motioncited by 7
  4. 2022Universal characteristics of deep neural network loss surfaces from random matrix theorycited by 4
  5. 2024Random Matrix Theory for Stochastic Gradient Descentcited by 0
  6. 2025Matricial Free Energy as a Gaussianizing Regularizer: Enhancing Autoencoders for Gaussian Code Generationcited by 0
  7. 2025Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theorycited by 0