# Rate Distortion For Model Compression: From Theory To Practice

arXiv: https://arxiv.org/abs/1810.06401 · HTML: https://mlsanity.com/paper/1810.06401/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2018.

## Built on by (most cited first, 15 of 15)

- [An Information-Theoretic Justification for Model Pruning](https://mlsanity.com/paper/2102.08329/) (2021)
- [Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning](https://mlsanity.com/paper/2105.04180/) (2021)
- [A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics](https://mlsanity.com/paper/2112.02834/) (2021)
- [High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve](https://mlsanity.com/paper/2309.07663/) (2023)
- [Fundamental Limitation of Semantic Communications: Neural Estimation for Rate-Distortion](https://mlsanity.com/paper/2401.01176/) (2024)
- [Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation](https://mlsanity.com/paper/2502.01717/) (2025)
- [Model-Aware Rate-Distortion Limits for Task-Oriented Source Coding](https://mlsanity.com/paper/2602.12866/) (2026)
- [Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms](https://mlsanity.com/paper/2202.11277/) (2022)
- [Rotation Invariant Quantization for Model Compression](https://mlsanity.com/paper/2303.03106/) (2023)
- [Sen2Pro: A Probabilistic Perspective to Sentence Embedding from Pre-trained Language Model](https://mlsanity.com/paper/2306.02247/) (2023)
- [BackSlash: Rate Constrained Optimized Training of Large Language Models](https://mlsanity.com/paper/2504.16968/) (2025)
- [Rate distortion comparison of a few gradient quantizers](https://mlsanity.com/paper/2108.09899/) (2021)
- [Approximability and Generalisation](https://mlsanity.com/paper/2203.07989/) (2022)
- [Exploring The Neural Burden In Pruned Models: An Insight Inspired By Neuroscience](https://mlsanity.com/paper/2407.16716/) (2024)
- [A Framework for Robust Lossy Compression of Heavy-Tailed Sources](https://mlsanity.com/paper/2411.08549/) (2024)

## Abstract (arXiv)

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are parameter pruning, parameter sharing (quantization), and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, namely pruning and quantization. We first study the fundamental limit for model compression via the rate distortion theory. We bring the rate distortion function from data compression to model compression to quantify this fundamental limit. We prove a lower bound for the rate distortion function and prove its achievability for linear models. Although this achievable compression scheme is intractable in practice, this analysis motivates a novel model compression framework. This framework provides a new objective function in model compression, which can be applied together with other classes of model compressor such as pruning or quantization. Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks. Empirically, we show that the proposed scheme improves upon the baseline in the compression-accuracy tradeoff.