# Pruning Deep Neural Networks via the Marchenko--Pastur Distribution

arXiv: https://arxiv.org/abs/2606.02608 · HTML: https://mlsanity.com/paper/2606.02608/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 23 May 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)
- [MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models](https://mlsanity.com/paper/2409.17481/) (2024)
- [Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks](https://mlsanity.com/paper/1901.08278/) (2019)
- [Random matrix analysis of deep neural network weight matrices](https://mlsanity.com/paper/2203.14661/) (2022)
- [CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models](https://mlsanity.com/paper/2210.09223/) (2022)
- [On Random Matrices Arising in Deep Neural Networks. Gaussian Case](https://mlsanity.com/paper/2001.06188/) (2020)
- [Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stopping](https://mlsanity.com/paper/2111.13331/) (2021)
- [Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework](https://mlsanity.com/paper/2411.18376/) (2024)
- [On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case](https://mlsanity.com/paper/2011.11439/) (2020)
- [Boundary between noise and information applied to filtering neural network weight matrices](https://mlsanity.com/paper/2206.03927/) (2022)
- [Heavy-Tailed Regularization of Weight Matrices in Deep Neural Networks](https://mlsanity.com/paper/2304.02911/) (2023)
- [On the Initialisation of Wide Low-Rank Feedforward Neural Networks](https://mlsanity.com/paper/2301.13710/) (2023)
- [UniPTS: A Unified Framework for Proficient Post-Training Sparsity](https://mlsanity.com/paper/2405.18810/) (2024)
- [Bootstrapping SparseFormers from Vision Foundation Models](https://mlsanity.com/paper/2312.01987/) (2023)

## Abstract (arXiv)

We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component R has small propagated logit effect Ls | R psi1(s) |_∞, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive L2-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge σ_+ gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16 2:4+ToMe reaches 83.41% top-1 (-1.70 pp from dense) at 59.81% sparse-execution MAC reduction, with 1.388× best-observed A40 native-2:4 backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives 2.705×. At structured sparsity, ViT-B/16 6:12 reaches 83.74%, ViT-L/16 8:16 dense+permutation reaches 85.33% (-0.51 pp), and ConvNeXtV2-Base 12:16 reaches 86.35% (-0.37 pp). For CNNs, ResNet50 8:16 dense+permutation reaches 75.87% (-0.26 pp), and ResNet152d CAST-conv+permutation reaches 81.33% (-1.53 pp) at ~50% MAC accounting with a 1.62× A40 im2col+2:4 sparse-GEMM audit.