Pruning Deep Neural Networks via the Marchenko--Pastur Distribution
Not on the map. Why?
Abstract (arXiv)
We study a Marchenko--Pastur (MP) random-matrix approach to pruning deep neural networks with very small post-pruning fine-tuning budgets. The main practical contribution is accuracy retention under short calibration and fine-tuning schedules, rather than a long post-pruning reoptimization pipeline. The theory gives deterministic data-path certificates: if the removed component R has small propagated logit effect Ls | R psi1(s) |_∞, pruning decreases an elastic-net objective and preserves samples whose dense margin exceeds twice the perturbation. The zero-budget case gives perfect pruning; a prune--restore extension models weight restoration inside a fixed sparse-execution pattern; and an additive L2-regularized model shows admissible random-like components vanish at the training limit, with persistent spikes stabilizing as the MP bulk collapses. Under iid-Gaussian sufficient conditions, the fitted MP edge σ_+ gives a high-probability layerwise budget signal. On ImageNet-1k, after only three distillation epochs, ViT-B/16 2:4+ToMe reaches 83.41% top-1 (-1.70 pp from dense) at 59.81% sparse-execution MAC reduction, with 1.388× best-observed A40 native-2:4 backend speedup for the same checkpoint and ToMe graph; a separate no-ToMe A100 endpoint gives 2.705×. At structured sparsity, ViT-B/16 6:12 reaches 83.74%, ViT-L/16 8:16 dense+permutation reaches 85.33% (-0.51 pp), and ConvNeXtV2-Base 12:16 reaches 86.35% (-0.37 pp). For CNNs, ResNet50 8:16 dense+permutation reaches 75.87% (-0.26 pp), and ResNet152d CAST-conv+permutation reaches 81.33% (-1.53 pp) at ~50% MAC accounting with a 1.62× A40 im2col+2:4 sparse-GEMM audit.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 23 May 2026.
Builds on
23 of 32 references.
References · 23
- 2017Attention Is All You Needcited by 59,932
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2022Token Merging: Your ViT But Fastercited by 995
- 2020What is the State of Neural Network Pruning?cited by 578
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
- 2019The State of Sparsity in Deep Neural Networkscited by 471
- 2021Accelerating Sparse Deep Neural Networkscited by 213
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
- 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
- 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
- 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
- 2019Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networkscited by 43
11 more
- 2022Random matrix analysis of deep neural network weight matricescited by 21
- 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
- 2020On Random Matrices Arising in Deep Neural Networks. Gaussian Casecited by 14
- 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
- 2020On Random Matrices Arising in Deep Neural Networks: General I.I.D. Casecited by 10
- 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 10
- 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
- 2023Heavy-Tailed Regularization of Weight Matrices in Deep Neural Networkscited by 7
- 2023On the Initialisation of Wide Low-Rank Feedforward Neural Networkscited by 5
- 2024UniPTS: A Unified Framework for Proficient Post-Training Sparsitycited by 3
- 2023Bootstrapping SparseFormers from Vision Foundation Modelscited by 1
Built on by
Not cited yet.