mlsanityData to 7 Oct 2026

Universal characteristics of deep neural network loss surfaces from random matrix theory

arXiv 2205.08601 ↗First posted 17 May 2022math-ph cond-mat.dis-nn cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper considers several aspects of random matrix universality in deep neural networks. Motivated by recent experimental work, we use universal properties of random matrices related to local statistics to derive practical implications for deep neural networks based on a realistic model of their Hessians. In particular we derive universal aspects of outliers in the spectra of deep neural networks and demonstrate the important role of random matrix local laws in popular pre-conditioning gradient descent algorithms. We also present insights into deep neural network loss surfaces from quite general arguments based on tools from statistical physics and random matrix theory.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 23 of its 69 references that are in the map. First posted on arXiv 17 May 2022.

Builds on

23 of 69 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Wide Residual Networkscited by 3,057
  5. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  6. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  7. 2014The Loss Surfaces of Multilayer Networkscited by 278
  8. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  9. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  10. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  11. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  12. 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
11 more
  1. 2018Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitionscited by 23
  2. 2022Random matrix analysis of deep neural network weight matricescited by 21
  3. 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
  4. 2020On Random Matrices Arising in Deep Neural Networks. Gaussian Casecited by 14
  5. 2020Beyond Random Matrix Theory for Deep Networkscited by 11
  6. 2020On Random Matrices Arising in Deep Neural Networks: General I.I.D. Casecited by 10
  7. 2014Explorations on high dimensional landscapescited by 8
  8. 2020The Loss Surfaces of Neural Networks with General Activation Functionscited by 7
  9. 2021Appearance of Random Matrix Theory in Deep Learningcited by 7
  10. 2021A spin-glass model for the loss surfaces of generative adversarial networkscited by 4
  11. 2020A Random Matrix Theory Approach to Damping in Deep Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  2. 2024Unifying Low Dimensional Spectra in Deep Learningcited by 11
  3. 2024Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularizationcited by 6
  4. 2026Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localizationcited by 0