# Random matrix analysis of deep neural network weight matrices

arXiv: https://arxiv.org/abs/2203.14661 · HTML: https://mlsanity.com/paper/2203.14661/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 105 references that are in the map.
- First posted on arXiv 28 Mar 2022.

## Builds on (most cited first, 30 of 32)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Two models of double descent for weak features](https://mlsanity.com/paper/1903.07571/) (2019)
- [Feature Learning in Infinite-Width Neural Networks](https://mlsanity.com/paper/2011.14522/) (2020)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel](https://mlsanity.com/paper/2010.15110/) (2020)
- [Deep learning generalizes because the parameter-function map is biased towards simple functions](https://mlsanity.com/paper/1805.08522/) (2018)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning](https://mlsanity.com/paper/1712.06559/) (2017)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [An analytic theory of generalization dynamics and transfer learning in deep linear networks](https://mlsanity.com/paper/1809.10374/) (2018)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [The Emergence of Spectral Universality in Deep Networks](https://mlsanity.com/paper/1802.09979/) (2018)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [On the saddle point problem for non-convex optimization](https://mlsanity.com/paper/1405.4604/) (2014)
- [Exponentially vanishing sub-optimal local minima in multilayer neural networks](https://mlsanity.com/paper/1702.05777/) (2017)

## Built on by (most cited first, 21 of 21)

- [RL Fine-Tuning Heals OOD Forgetting in SFT](https://mlsanity.com/paper/2509.12235/) (2025)
- [Spectral Evolution and Invariance in Linear-width Neural Networks](https://mlsanity.com/paper/2211.06506/) (2022)
- [Small Singular Values Matter: A Random Matrix Analysis of Transformer Models](https://mlsanity.com/paper/2410.17770/) (2024)
- [A Rainbow in Deep Network Black Boxes](https://mlsanity.com/paper/2305.18512/) (2023)
- [Asymptotics of Learning with Deep Structured (Random) Features](https://mlsanity.com/paper/2402.13999/) (2024)
- [Models of Heavy-Tailed Mechanistic Universality](https://mlsanity.com/paper/2506.03470/) (2025)
- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [The Underlying Scaling Laws and Universal Statistical Structure of Complex Datasets](https://mlsanity.com/paper/2306.14975/) (2023)
- [Universal characteristics of deep neural network loss surfaces from random matrix theory](https://mlsanity.com/paper/2205.08601/) (2022)
- [LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2502.06820/) (2025)
- [Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario](https://mlsanity.com/paper/2604.18450/) (2026)
- [Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining](https://mlsanity.com/paper/2605.20296/) (2026)
- [The Universal Statistical Structure and Scaling Laws of Chaos and Turbulence](https://mlsanity.com/paper/2311.01358/) (2023)
- [Task complexity shapes internal representations and robustness in neural networks](https://mlsanity.com/paper/2508.05463/) (2025)
- [Randomness and signal propagation in physics-informed neural networks (PINNs): A neural PDE perspective](https://mlsanity.com/paper/2509.18131/) (2025)
- [Spectral Signatures of Data Quality: Eigenvalue Tail Index as a Diagnostic for Label Noise in Neural Networks](https://mlsanity.com/paper/2603.27885/) (2026)
- [Eigenvectors of Experts are Training-free Non-collapsing Routers](https://mlsanity.com/paper/2605.30992/) (2026)
- [Doubly Stochastic Models: Learning with Unbiased Label Noises and Inference Stability](https://mlsanity.com/paper/2304.00320/) (2023)
- [Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory](https://mlsanity.com/paper/2512.12911/) (2025)
- [Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration](https://mlsanity.com/paper/2604.02659/) (2026)
- [Pruning Deep Neural Networks via the Marchenko--Pastur Distribution](https://mlsanity.com/paper/2606.02608/) (2026)

## Abstract (arXiv)

Neural networks have been used successfully in a variety of fields, which has led to a great deal of interest in developing a theoretical understanding of how they store the information needed to perform a particular task. We study the weight matrices of trained deep neural networks using methods from random matrix theory (RMT) and show that the statistics of most of the singular values follow universal RMT predictions. This suggests that they are random and do not contain system specific information, which we investigate further by comparing the statistics of eigenvector entries to the universal Porter-Thomas distribution. We find that for most eigenvectors the hypothesis of randomness cannot be rejected, and that only eigenvectors belonging to the largest singular values deviate from the RMT prediction, indicating that they may encode learned information. In addition, a comparison with RMT predictions also allows to distinguish networks trained in different learning regimes - from lazy to rich learning. We analyze the spectral distribution of the large singular values using the Hill estimator and find that the distribution cannot in general be characterized by a tail index, i.e. is not of power law type.