# Neural Networks Optimally Compress the Sawbridge

arXiv: https://arxiv.org/abs/2011.05065 · HTML: https://mlsanity.com/paper/2011.05065/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2020.

## Built on by (most cited first, 11 of 11)

- [An Introduction to Neural Data Compression](https://mlsanity.com/paper/2202.06533/) (2022)
- [On the advantages of stochastic encoders](https://mlsanity.com/paper/2102.09270/) (2021)
- [The Rate-Distortion-Perception Tradeoff: The Role of Common Randomness](https://mlsanity.com/paper/2202.04147/) (2022)
- [Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding](https://mlsanity.com/paper/2204.01612/) (2022)
- [Approaching Rate-Distortion Limits in Neural Compression with Lattice Transform Coding](https://mlsanity.com/paper/2403.07320/) (2024)
- [Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods](https://mlsanity.com/paper/2212.13468/) (2022)
- [Do Neural Networks Compress Manifolds Optimally?](https://mlsanity.com/paper/2205.08518/) (2022)
- [DeCompress: Denoising via Neural Compression](https://mlsanity.com/paper/2503.22015/) (2025)
- [Out-of-Distribution Robustness in Deep Learning Compression](https://mlsanity.com/paper/2110.07007/) (2021)
- [Federated Representation Learning via Maximal Coding Rate Reduction](https://mlsanity.com/paper/2210.00299/) (2022)
- [On One-Bit Quantization](https://mlsanity.com/paper/2202.05292/) (2022)

## Abstract (arXiv)

Neural-network-based compressors have proven to be remarkably effective at compressing sources, such as images, that are nominally high-dimensional but presumed to be concentrated on a low-dimensional manifold. We consider a continuous-time random process that models an extreme version of such a source, wherein the realizations fall along a one-dimensional "curve" in function space that has infinite-dimensional linear span. We precisely characterize the optimal entropy-distortion tradeoff for this source and show numerically that it is achieved by neural-network-based compressors trained via stochastic gradient descent. In contrast, we show both analytically and experimentally that compressors based on the classical Karhunen-Lo`eve transform are highly suboptimal at high rates.