Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks
Not on the map. Why?
Abstract (arXiv)
Deep neural networks have achieved state-of-the-art performance across numerous applications, but their high memory and computational demands present significant challenges, particularly in resource-constrained environments. Model compression techniques, such as low-rank approximation, offer a promising solution by reducing the size and complexity of these networks while only minimally sacrificing accuracy. In this paper, we develop an analytical framework for data-driven post-training low-rank compression. We prove three recovery theorems under progressively weaker assumptions about the approximate low-rank structure of activations, modeling deviations via noise. Our results represent a step toward explaining why data-driven low-rank compression methods outperform data-agnostic approaches and towards theoretically grounded compression algorithms that reduce inference costs while maintaining performance.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 64 references that are in the map. First posted on arXiv 4 Feb 2025.
Builds on
29 of 64 references.
References · 29
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2020Reformer: The Efficient Transformercited by 1,658
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
- 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
- 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
- 2023A Survey on Model Compression for Large Language Modelscited by 263
- 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
17 more
- 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
- 2020An Overview of Neural Network Compressioncited by 59
- 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
- 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
- 2019Low-Rank Matrix Completion: A Contemporary Surveycited by 43
- 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
- 2014On Tensor Completion via Nuclear Norm Minimizationcited by 32
- 2022Post-training Quantization for Neural Networks with Provable Guaranteescited by 32
- 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
- 2013ROP: Matrix recovery via rank-one projectionscited by 18
- 2023Neural (Tangent Kernel) Collapsecited by 18
- 2023Tensor Decomposition for Model Reduction in Neural Networks: A Reviewcited by 14
- 2020Inference for Low-rank Tensors -- No Need to Debiascited by 12
- 2020Low-Rank and Sparse Enhanced Tucker Decomposition for Tensor Completioncited by 4
- 2021Data-Driven Low-Rank Neural Network Compressioncited by 3
- 2022Improved Projection Learning for Lower Dimensional Feature Mapscited by 2
- 2015Low-Rank Matrix Recovery from Row-and-Column Affine Measurementscited by 1
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.