# Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks

arXiv: https://arxiv.org/abs/2502.02766 · HTML: https://mlsanity.com/paper/2502.02766/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 64 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 29 of 29)

- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Understanding Diffusion Models: A Unified Perspective](https://mlsanity.com/paper/2208.11970/) (2022)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [An Overview of Neural Network Compression](https://mlsanity.com/paper/2006.03669/) (2020)
- [Implicit Regularization Towards Rank Minimization in ReLU Networks](https://mlsanity.com/paper/2201.12760/) (2022)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Low-Rank Matrix Completion: A Contemporary Survey](https://mlsanity.com/paper/1907.11705/) (2019)
- [Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorization](https://mlsanity.com/paper/2203.02839/) (2022)
- [Post-training Quantization for Neural Networks with Provable Guarantees](https://mlsanity.com/paper/2201.11113/) (2022)
- [On Tensor Completion via Nuclear Norm Minimization](https://mlsanity.com/paper/1405.1773/) (2014)
- [HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers](https://mlsanity.com/paper/2302.09632/) (2023)
- [ROP: Matrix recovery via rank-one projections](https://mlsanity.com/paper/1310.5791/) (2013)
- [Neural (Tangent Kernel) Collapse](https://mlsanity.com/paper/2305.16427/) (2023)
- [Tensor Decomposition for Model Reduction in Neural Networks: A Review](https://mlsanity.com/paper/2304.13539/) (2023)
- [Inference for Low-rank Tensors -- No Need to Debias](https://mlsanity.com/paper/2012.14844/) (2020)
- [Low-Rank and Sparse Enhanced Tucker Decomposition for Tensor Completion](https://mlsanity.com/paper/2010.00359/) (2020)
- [Data-Driven Low-Rank Neural Network Compression](https://mlsanity.com/paper/2107.05787/) (2021)
- [Improved Projection Learning for Lower Dimensional Feature Maps](https://mlsanity.com/paper/2210.15170/) (2022)
- [Low-Rank Matrix Recovery from Row-and-Column Affine Measurements](https://mlsanity.com/paper/1505.06292/) (2015)

## Built on by (most cited first, 3 of 3)

- [Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronos](https://mlsanity.com/paper/2508.04853/) (2025)
- [Beacon: Post-Training Quantization with Integrated Grid Selection](https://mlsanity.com/paper/2508.20293/) (2025)
- [GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation](https://mlsanity.com/paper/2606.01412/) (2026)

## Abstract (arXiv)

Deep neural networks have achieved state-of-the-art performance across numerous applications, but their high memory and computational demands present significant challenges, particularly in resource-constrained environments. Model compression techniques, such as low-rank approximation, offer a promising solution by reducing the size and complexity of these networks while only minimally sacrificing accuracy. In this paper, we develop an analytical framework for data-driven post-training low-rank compression. We prove three recovery theorems under progressively weaker assumptions about the approximate low-rank structure of activations, modeling deviations via noise. Our results represent a step toward explaining why data-driven low-rank compression methods outperform data-agnostic approaches and towards theoretically grounded compression algorithms that reduce inference costs while maintaining performance.