# Model Preserving Compression for Neural Networks

arXiv: https://arxiv.org/abs/2108.00065 · HTML: https://mlsanity.com/paper/2108.00065/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 80 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 30 of 42)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [A comparative study of fairness-enhancing interventions in machine learning](https://mlsanity.com/paper/1802.04422/) (2018)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Data-Driven Sparse Structure Selection for Deep Neural Networks](https://mlsanity.com/paper/1707.01213/) (2017)

## Built on by (most cited first, 9 of 9)

- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [Model-Preserving Adaptive Rounding](https://mlsanity.com/paper/2505.22988/) (2025)
- [STAT: Shrinking Transformers After Training](https://mlsanity.com/paper/2406.00061/) (2024)
- [CURing Large Models: Compression via CUR Decomposition](https://mlsanity.com/paper/2501.04211/) (2025)
- [Neural Weight Compression for Language Models](https://mlsanity.com/paper/2510.11234/) (2025)
- [Pruning AMR: Efficient Visualization of Implicit Neural Representations via Weight Matrix Analysis](https://mlsanity.com/paper/2512.02967/) (2025)
- [Accelerating Attention with Basis Decomposition](https://mlsanity.com/paper/2510.01718/) (2025)
- [Optimizing Tensor Train Decomposition in DNNs for RISC-V Architectures Using Design Space Exploration and Compiler Optimizations](https://mlsanity.com/paper/2602.01996/) (2026)
- [Lottery Tickets Are Not Deployment Tickets](https://mlsanity.com/paper/2607.27031/) (2026)

## Abstract (arXiv)

After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it is desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy or preserve robustness), maintain the network's structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultaneously satisfy these criteria unicodex2014 we introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance on a variety of tasks, models, and datasets unicodex2014 from simple one-hidden-layer networks to deep networks on ImageNet.