# Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory

arXiv: https://arxiv.org/abs/2512.12911 · HTML: https://mlsanity.com/paper/2512.12911/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 15 Dec 2025.

## Builds on (most cited first, 14 of 14)

- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [Small Singular Values Matter: A Random Matrix Analysis of Transformer Models](https://mlsanity.com/paper/2410.17770/) (2024)
- [Estimation of the number of spiked eigenvalues in a covariance matrix by bulk eigenvalue matching analysis](https://mlsanity.com/paper/2006.00436/) (2020)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [Grokking and Generalization Collapse: Insights from textttHTSR theory](https://mlsanity.com/paper/2506.04434/) (2025)
- [Appearance of Random Matrix Theory in Deep Learning](https://mlsanity.com/paper/2102.06740/) (2021)
- [Enhancing Accuracy in Deep Learning Using Random Matrix Theory](https://mlsanity.com/paper/2310.03165/) (2023)
- [Pruning Deep Neural Networks via a Combination of the Marchenko-Pastur Distribution and Regularization](https://mlsanity.com/paper/2503.01922/) (2025)

## Abstract (arXiv)

This study evaluates thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices. Each weight matrix is modeled as the sum of signal and noise matrices. The low-rank approximation is obtained by removing noise-related singular values using a threshold based on random matrix theory. To assess the adequacy of this threshold, we propose an evaluation metric based on the cosine similarity between the singular vectors of the signal and original weight matrices. The proposed metric is used in numerical experiments to compare two threshold estimation methods.