# Model compression as constrained optimization, with application to neural nets. Part II: quantization

arXiv: https://arxiv.org/abs/1707.04319 · HTML: https://mlsanity.com/paper/1707.04319/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2017.

## Built on by (most cited first, 8 of 8)

- [Finding the Task-Optimal Low-Bit Sub-Distribution in Deep Neural Networks](https://mlsanity.com/paper/2112.15139/) (2021)
- [Hyperspherical Quantization: Toward Smaller and More Accurate Models](https://mlsanity.com/paper/2212.12653/) (2022)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee](https://mlsanity.com/paper/2303.06815/) (2023)
- [A Discrepancy-Based Perspective on Dataset Condensation](https://mlsanity.com/paper/2509.10367/) (2025)
- [Pruning Ternary Quantization](https://mlsanity.com/paper/2107.10998/) (2021)
- [Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression](https://mlsanity.com/paper/2111.10320/) (2021)
- [Joint Quantization and Pruning Neural Networks Approach: A Case Study on FSO Receivers](https://mlsanity.com/paper/2506.20084/) (2025)

## Abstract (arXiv)

We consider the problem of deep neural net compression by quantization: given a large, reference net, we want to quantize its real-valued weights using a codebook with K entries so that the training loss of the quantized net is minimal. The codebook can be optimally learned jointly with the net, or fixed, as for binarization or ternarization approaches. Previous work has quantized the weights of the reference net, or incorporated rounding operations in the backpropagation algorithm, but this has no guarantee of converging to a loss-optimal, quantized net. We describe a new approach based on the recently proposed framework of model compression as constrained optimization citepCarreir17a. This results in a simple iterative "learning-compression" algorithm, which alternates a step that learns a net of continuous weights with a step that quantizes (or binarizes/ternarizes) the weights, and is guaranteed to converge to local optimum of the loss for quantized nets. We develop algorithms for an adaptive codebook or a (partially) fixed codebook. The latter includes binarization, ternarization, powers-of-two and other important particular cases. We show experimentally that we can achieve much higher compression rates than previous quantization work (even using just 1 bit per weight) with negligible loss degradation.