mlsanityData to 7 Oct 2026

Enhancing Accuracy in Deep Learning Using Random Matrix Theory

arXiv 2310.03165 ↗First posted 4 Oct 2023cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

We explore the applications of random matrix theory (RMT) in the training of deep neural networks (DNNs), focusing on layer pruning that is reducing the number of DNN parameters (weights). Our numerical results show that this pruning leads to a drastic reduction of parameters while not reducing the accuracy of DNNs and CNNs. Moreover, pruning the fully connected DNNs actually increases the accuracy and decreases the variance for random initializations. Our numerics indicate that this enhancement in accuracy is due to the simplification of the loss landscape. We next provide rigorous mathematical underpinning of these numerical results by proving the RMT-based Pruning Theorem. Our results offer valuable insights into the practical application of RMT for the creation of more efficient and accurate deep-learning models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 17 of its 49 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

17 of 49 references.

References · 17

  1. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  4. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  5. 2014The Loss Surfaces of Multilayer Networkscited by 278
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  7. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  8. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  9. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  10. 2018Trained Rank Pruning for Efficient Deep Neural Networkscited by 23
  11. 2020Estimation of the number of spiked eigenvalues in a covariance matrix by bulk eigenvalue matching analysiscited by 15
  12. 2021Entrywise Estimation of Singular Vectors of Low-Rank Matrices with Heteroskedasticity and Dependencecited by 12
5 more
  1. 2018Asymmetry Helps: Eigenvalue and Eigenvector Analyses of Asymmetrically Perturbed Low-Rank Matricescited by 10
  2. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  3. 2018Matrices with Gaussian noise: optimal estimates for singular subspace perturbationcited by 6
  4. 2019Matrix denoising for weighted loss functions and heterogeneous signalscited by 6
  5. 2023Deep Learning Weight Pruning with RMT-SVD: Increasing Accuracy and Reducing Overfittingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Random weights of DNNs and emergence of fixed pointscited by 1
  2. 2025Early-stopping for Transformer model trainingcited by 1
  3. 2025Asymptotic behavior of eigenvalues of large rank perturbations of large random matricescited by 0
  4. 2025Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theorycited by 0
  5. 2026Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alonecited by 0