mlsanityData to 14 Oct 2026

Boundary between noise and information applied to filtering neural network weight matrices

arXiv 2206.03927 ↗First posted 8 Jun 2022cond-mat.dis-nn cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 23 of its 63 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

23 of 63 references.

References · 23

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2021Scaling Vision Transformerscited by 999
  3. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  4. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  5. 2014Qualitatively characterizing neural network optimization problemscited by 188
  6. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  7. 2017Generalization in Deep Learningcited by 146
  8. 2018Fast inference of deep neural networks in FPGAs for particle physicscited by 133
  9. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  10. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  11. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  12. 2017A Random Matrix Approach to Neural Networkscited by 80
11 more
  1. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  2. 2017Compression-aware Training of Deep Networkscited by 46
  3. 2017Mutual Information, Neural Networks and the Renormalization Groupcited by 33
  4. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  5. 2022Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometrycited by 28
  6. 2018Trained Rank Pruning for Efficient Deep Neural Networkscited by 23
  7. 2022Random matrix analysis of deep neural network weight matricescited by 21
  8. 2017Exponentially vanishing sub-optimal local minima in multilayer neural networkscited by 19
  9. 2020Beyond Random Matrix Theory for Deep Networkscited by 11
  10. 2021Appearance of Random Matrix Theory in Deep Learningcited by 7
  11. 2021Physics-informed generative neural network: an application to troposphere temperature predictioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 2; 2026: 3.

9 papers.

  1. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
  2. 2023Enhancing Accuracy in Deep Learning Using Random Matrix Theorycited by 5
  3. 2026Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenariocited by 3
  4. 2026Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retrainingcited by 3
  5. 2023Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfittingcited by 2
  6. 2025Task complexity shapes internal representations and robustness in neural networkscited by 2
  7. 2022Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Conditioncited by 0
  8. 2025Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theorycited by 0
  9. 2026Pruning Deep Neural Networks via the Marchenko--Pastur Distributioncited by 0