mlsanityData to 7 Oct 2026

Random matrix analysis of deep neural network weight matrices

arXiv 2203.14661 ↗First posted 28 Mar 2022cond-mat.dis-nn cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

Neural networks have been used successfully in a variety of fields, which has led to a great deal of interest in developing a theoretical understanding of how they store the information needed to perform a particular task. We study the weight matrices of trained deep neural networks using methods from random matrix theory (RMT) and show that the statistics of most of the singular values follow universal RMT predictions. This suggests that they are random and do not contain system specific information, which we investigate further by comparing the statistics of eigenvector entries to the universal Porter-Thomas distribution. We find that for most eigenvectors the hypothesis of randomness cannot be rejected, and that only eigenvectors belonging to the largest singular values deviate from the RMT prediction, indicating that they may encode learned information. In addition, a comparison with RMT predictions also allows to distinguish networks trained in different learning regimes - from lazy to rich learning. We analyze the spectral distribution of the large singular values using the Hill estimator and find that the distribution cannot in general be characterized by a tail index, i.e. is not of power law type.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 32 of its 105 references that are in the map. First posted on arXiv 28 Mar 2022.

Builds on

32 of 105 references.

References · 32

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2021Scaling Vision Transformerscited by 993
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2018On Lazy Training in Differentiable Programmingcited by 671
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  12. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
20 more
  1. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  2. 2019Two models of double descent for weak featurescited by 194
  3. 2014Qualitatively characterizing neural network optimization problemscited by 186
  4. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
  5. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  6. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  7. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  8. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  9. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  10. 2017Generalization in Deep Learningcited by 146
  11. 2018Fast inference of deep neural networks in FPGAs for particle physicscited by 132
  12. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  13. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  14. 2017A Random Matrix Approach to Neural Networkscited by 80
  15. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  16. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  17. 2014On the saddle point problem for non-convex optimizationcited by 19
  18. 2017Exponentially vanishing sub-optimal local minima in multilayer neural networkscited by 19
  19. 2020Beyond Random Matrix Theory for Deep Networkscited by 11
  20. 2021Physics-informed generative neural network: an application to troposphere temperature predictioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 2; 2025: 6; 2026: 6.

21 papers.

Sort:

By area:

  1. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  2. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 27
  3. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  4. 2023A Rainbow in Deep Network Black Boxescited by 18
  5. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  6. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  7. 2023The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasetscited by 9
  8. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  9. 2022Universal characteristics of deep neural network loss surfaces from random matrix theorycited by 4
  10. 2025LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuningcited by 4
  11. 2023The Universal Statistical Structure and Scaling Laws of Chaos and Turbulencecited by 2
  12. 2025Task complexity shapes internal representations and robustness in neural networkscited by 2
9 more
  1. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 2
  2. 2026Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retrainingcited by 2
  3. 2025Randomness and signal propagation in physics-informed neural networks (PINNs): A neural PDE perspectivecited by 1
  4. 2026Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networkscited by 1
  5. 2026Eigenvectors of Experts are Training-free Non-collapsing Routerscited by 1
  6. 2023Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stabilitycited by 0
  7. 2025Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theorycited by 0
  8. 2026Low-Rank Compression of Pretrained Models via Randomized Subspace Iterationcited by 0
  9. 2026Pruning Deep Neural Networks via the Marchenko--Pastur Distributioncited by 0