mlsanityData to 7 Oct 2026

Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory

arXiv 2512.12911 ↗First posted 15 Dec 2025stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This study evaluates thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices. Each weight matrix is modeled as the sum of signal and noise matrices. The low-rank approximation is obtained by removing noise-related singular values using a threshold based on random matrix theory. To assess the adequacy of this threshold, we propose an evaluation metric based on the cosine similarity between the singular vectors of the signal and original weight matrices. The proposed metric is used in numerical experiments to compare two threshold estimation methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

14 of 30 references.

References · 14

  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  2. 2017A Closer Look at Memorization in Deep Networkscited by 936
  3. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  4. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  5. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  6. 2022Random matrix analysis of deep neural network weight matricescited by 21
  7. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 20
  8. 2020Estimation of the number of spiked eigenvalues in a covariance matrix by bulk eigenvalue matching analysiscited by 15
  9. 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
  10. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  11. 2025Grokking and Generalization Collapse: Insights from textttHTSR theorycited by 8
  12. 2021Appearance of Random Matrix Theory in Deep Learningcited by 7
2 more
  1. 2023Enhancing Accuracy in Deep Learning Using Random Matrix Theorycited by 5
  2. 2025Pruning Deep Neural Networks via a Combination of the Marchenko-Pastur Distribution and Regularizationcited by 3

Built on by

Not cited yet.