# Boundary between noise and information applied to filtering neural network weight matrices

arXiv: https://arxiv.org/abs/2206.03927 · HTML: https://mlsanity.com/paper/2206.03927/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 63 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 23 of 23)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning](https://mlsanity.com/paper/1712.06559/) (2017)
- [Generalization in Deep Learning](https://mlsanity.com/paper/1710.05468/) (2017)
- [Fast inference of deep neural networks in FPGAs for particle physics](https://mlsanity.com/paper/1804.06913/) (2018)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [Robust Training under Label Noise by Over-parameterization](https://mlsanity.com/paper/2202.14026/) (2022)
- [An analytic theory of generalization dynamics and transfer learning in deep linear networks](https://mlsanity.com/paper/1809.10374/) (2018)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [The Emergence of Spectral Universality in Deep Networks](https://mlsanity.com/paper/1802.09979/) (2018)
- [Compression-aware Training of Deep Networks](https://mlsanity.com/paper/1711.02638/) (2017)
- [Mutual Information, Neural Networks and the Renormalization Group](https://mlsanity.com/paper/1704.06279/) (2017)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Trained Rank Pruning for Efficient Deep Neural Networks](https://mlsanity.com/paper/1812.02402/) (2018)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [Exponentially vanishing sub-optimal local minima in multilayer neural networks](https://mlsanity.com/paper/1702.05777/) (2017)
- [Beyond Random Matrix Theory for Deep Networks](https://mlsanity.com/paper/2006.07721/) (2020)
- [Appearance of Random Matrix Theory in Deep Learning](https://mlsanity.com/paper/2102.06740/) (2021)
- [Physics-informed generative neural network: an application to troposphere temperature prediction](https://mlsanity.com/paper/2107.06991/) (2021)

## Built on by (most cited first, 9 of 9)

- [Small Singular Values Matter: A Random Matrix Analysis of Transformer Models](https://mlsanity.com/paper/2410.17770/) (2024)
- [Enhancing Accuracy in Deep Learning Using Random Matrix Theory](https://mlsanity.com/paper/2310.03165/) (2023)
- [Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario](https://mlsanity.com/paper/2604.18450/) (2026)
- [Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining](https://mlsanity.com/paper/2605.20296/) (2026)
- [Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfitting](https://mlsanity.com/paper/2303.08986/) (2023)
- [Task complexity shapes internal representations and robustness in neural networks](https://mlsanity.com/paper/2508.05463/) (2025)
- [Stability of Accuracy for the Training of DNNs Via the Uniform Doubling Condition](https://mlsanity.com/paper/2210.08415/) (2022)
- [Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory](https://mlsanity.com/paper/2512.12911/) (2025)
- [Pruning Deep Neural Networks via the Marchenko--Pastur Distribution](https://mlsanity.com/paper/2606.02608/) (2026)

## Abstract (arXiv)

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.