# Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding

arXiv: https://arxiv.org/abs/2204.01612 · HTML: https://mlsanity.com/paper/2204.01612/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 45 references that are in the map.
- First posted on arXiv 4 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Compressing Images by Encoding Their Latent Representations with Relative Entropy Coding](https://mlsanity.com/paper/2010.01185/) (2020)
- [Algorithms for the Communication of Samples](https://mlsanity.com/paper/2110.12805/) (2021)
- [Evaluating Lossy Compression Rates of Deep Generative Models](https://mlsanity.com/paper/2008.06653/) (2020)
- [Towards Empirical Sandwich Bounds on the Rate-Distortion Function](https://mlsanity.com/paper/2111.12166/) (2021)
- [Neural Networks Optimally Compress the Sawbridge](https://mlsanity.com/paper/2011.05065/) (2020)
- [Exact Rate-Distortion in Autoencoders via Echo Noise](https://mlsanity.com/paper/1904.07199/) (2019)

## Built on by (most cited first, 9 of 9)

- [Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models](https://mlsanity.com/paper/2407.15504/) (2024)
- [Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding](https://mlsanity.com/paper/2403.07320/) (2024)
- [Estimating the Rate-Distortion Function by Wasserstein Gradient Descent](https://mlsanity.com/paper/2310.18908/) (2023)
- [Fundamental Limitation of Semantic Communications: Neural Estimation for Rate-Distortion](https://mlsanity.com/paper/2401.01176/) (2024)
- [Mathematical Formalism for Memory Compression in Selective State Space Models](https://mlsanity.com/paper/2410.03158/) (2024)
- [Optimal Neural Compressors for the Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/2503.17558/) (2025)
- [Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification](https://mlsanity.com/paper/2601.20742/) (2026)
- [On a Relation Between the Rate-Distortion Function and Optimal Transport](https://mlsanity.com/paper/2307.00246/) (2023)
- [PrismQuant: Rate-Distortion-Optimal Vector Quantization for Gaussian-Mixture Sources](https://mlsanity.com/paper/2605.15507/) (2026)

## Abstract (arXiv)

A fundamental question in designing lossy data compression schemes is how well one can do in comparison with the rate-distortion function, which describes the known theoretical limits of lossy compression. Motivated by the empirical success of deep neural network (DNN) compressors on large, real-world data, we investigate methods to estimate the rate-distortion function on such data, which would allow comparison of DNN compressors with optimality. While one could use the empirical distribution of the data and apply the Blahut-Arimoto algorithm, this approach presents several computational challenges and inaccuracies when the datasets are large and high-dimensional, such as the case of modern image datasets. Instead, we re-formulate the rate-distortion objective, and solve the resulting functional optimization problem using neural networks. We apply the resulting rate-distortion estimator, called NERD, on popular image datasets, and provide evidence that NERD can accurately estimate the rate-distortion function. Using our estimate, we show that the rate-distortion achievable by DNN compressors are within several bits of the rate-distortion function for real-world datasets. Additionally, NERD provides access to the rate-distortion achieving channel, as well as samples from its output marginal. Therefore, using recent results in reverse channel coding, we describe how NERD can be used to construct an operational one-shot lossy compression scheme with guarantees on the achievable rate and distortion. Experimental results demonstrate competitive performance with DNN compressors.