# Enhancing Accuracy in Deep Learning Using Random Matrix Theory

arXiv: https://arxiv.org/abs/2310.03165 · HTML: https://mlsanity.com/paper/2310.03165/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition](https://mlsanity.com/paper/1503.02101/) (2015)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification](https://mlsanity.com/paper/2004.09031/) (2020)
- [Trained Rank Pruning for Efficient Deep Neural Networks](https://mlsanity.com/paper/1812.02402/) (2018)
- [Estimation of the number of spiked eigenvalues in a covariance matrix by bulk eigenvalue matching analysis](https://mlsanity.com/paper/2006.00436/) (2020)
- [Entrywise Estimation of Singular Vectors of Low-Rank Matrices with Heteroskedasticity and Dependence](https://mlsanity.com/paper/2105.13346/) (2021)
- [Asymmetry Helps: Eigenvalue and Eigenvector Analyses of Asymmetrically Perturbed Low-Rank Matrices](https://mlsanity.com/paper/1811.12804/) (2018)
- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [Matrices with Gaussian noise: optimal estimates for singular subspace perturbation](https://mlsanity.com/paper/1803.00679/) (2018)
- [Matrix denoising for weighted loss functions and heterogeneous signals](https://mlsanity.com/paper/1902.09474/) (2019)
- [Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfitting](https://mlsanity.com/paper/2303.08986/) (2023)

## Built on by (most cited first, 5 of 5)

- [Random weights of DNNs and emergence of fixed points](https://mlsanity.com/paper/2501.04182/) (2025)
- [Early-stopping for Transformer model training](https://mlsanity.com/paper/2510.16074/) (2025)
- [Asymptotic behavior of eigenvalues of large rank perturbations of large random matrices](https://mlsanity.com/paper/2507.12182/) (2025)
- [Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory](https://mlsanity.com/paper/2512.12911/) (2025)
- [Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone](https://mlsanity.com/paper/2609.23087/) (2026)

## Abstract (arXiv)

We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that is reducing the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and CNNs. Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models.