# Model compression as constrained optimization, with application to neural nets. Part I: general framework

arXiv: https://arxiv.org/abs/1707.01209 · HTML: https://mlsanity.com/paper/1707.01209/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2017.

## Built on by (most cited first, 6 of 6)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [When in Doubt, Summon the Titans: Efficient Inference with Large Models](https://mlsanity.com/paper/2110.10305/) (2021)
- [Model compression as constrained optimization, with application to neural nets. Part V: combining compressions](https://mlsanity.com/paper/2107.04380/) (2021)
- [On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee](https://mlsanity.com/paper/2303.06815/) (2023)
- [A Discrepancy-Based Perspective on Dataset Condensation](https://mlsanity.com/paper/2509.10367/) (2025)
- [Search-Optimized Quantization in Biomedical Ontology Alignment](https://mlsanity.com/paper/2507.13742/) (2025)

## Abstract (arXiv)

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quantization, low-rank decomposition, pruning, lossless compression and others. Then, we give a general algorithm to optimize this nonconvex problem based on the augmented Lagrangian and alternating optimization. This results in a "learning-compression" algorithm, which alternates a learning step of the uncompressed model, independent of the compression type, with a compression step of the model parameters, independent of the learning task. This simple, efficient algorithm is guaranteed to find the best compressed model for the task in a local sense under standard assumptions. We present separately in several companion papers the development of this general framework into specific algorithms for model compression based on quantization, pruning and other variations, including experimental results on compressing neural nets and other models.