# On Perceptual Lossy Compression: The Cost of Perceptual Reconstruction and An Optimal Training Framework

arXiv: https://arxiv.org/abs/2106.02782 · HTML: https://mlsanity.com/paper/2106.02782/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 5 Jun 2021.

## Builds on (most cited first, 30 of 36)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks](https://mlsanity.com/paper/1809.00219/) (2018)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Learning a No-Reference Quality Metric for Single-Image Super-Resolution](https://mlsanity.com/paper/1612.05890/) (2016)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [Improved Lossy Image Compression with Priming and Spatially Adaptive Bit Rates for Recurrent Networks](https://mlsanity.com/paper/1703.10114/) (2017)
- [Learning Convolutional Networks for Content-weighted Image Compression](https://mlsanity.com/paper/1703.10553/) (2017)
- [End-to-end optimization of nonlinear transform codes for perceptual quality](https://mlsanity.com/paper/1607.05006/) (2016)

## Built on by (most cited first, 27 of 27)

- [Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models](https://mlsanity.com/paper/2301.11189/) (2023)
- [Lossy Compression with Gaussian Diffusion](https://mlsanity.com/paper/2206.08889/) (2022)
- [Universal Rate-Distortion-Perception Representations for Lossy Compression](https://mlsanity.com/paper/2106.10311/) (2021)
- [StableCodec: Taming One-Step Diffusion for Extreme Image Compression](https://mlsanity.com/paper/2506.21977/) (2025)
- [Idempotence and Perceptual Image Compression](https://mlsanity.com/paper/2401.08920/) (2024)
- [Optimal Transport for Unsupervised Denoising Learning](https://mlsanity.com/paper/2108.02574/) (2021)
- [EGIC: Enhanced Low-Bit-Rate Generative Image Compression Guided by Semantic Segmentation](https://mlsanity.com/paper/2309.03244/) (2023)
- [Optimally Controllable Perceptual Lossy Compression](https://mlsanity.com/paper/2206.10082/) (2022)
- [Generative Latent Video Compression](https://mlsanity.com/paper/2510.09987/) (2025)
- [On the Choice of Perception Loss Function for Learned Video Compression](https://mlsanity.com/paper/2305.19301/) (2023)
- [Rate-Distortion-Perception Tradeoff Based on the Conditional-Distribution Perception Measure](https://mlsanity.com/paper/2401.12207/) (2024)
- [Ultra-Low Bitrate Perceptual Image Compression with Shallow Encoder](https://mlsanity.com/paper/2512.12229/) (2025)
- [Conditional Perceptual Quality Preserving Image Compression](https://mlsanity.com/paper/2308.08154/) (2023)
- [Output-Constrained Lossy Source Coding With Application to Rate-Distortion-Perception Theory](https://mlsanity.com/paper/2403.14849/) (2024)
- [Gaussian Rate-Distortion-Perception Coding and Entropy-Constrained Scalar Quantization](https://mlsanity.com/paper/2409.02388/) (2024)
- [Synonymous Variational Inference for Perceptual Image Compression](https://mlsanity.com/paper/2505.22438/) (2025)
- [Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior](https://mlsanity.com/paper/2511.22549/) (2025)
- [Point Cloud Compression via Constrained Optimal Transport](https://mlsanity.com/paper/2403.08236/) (2024)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)
- [Optimal Neural Compressors for the Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/2503.17558/) (2025)
- [Rate-Distortion-Perception Theory for the Quadratic Wasserstein Space](https://mlsanity.com/paper/2504.17236/) (2025)
- [Restoration of the JPEG Maximum Lossy Compressed Face Images with Hourglass Block based on Early Stopping Discriminator](https://mlsanity.com/paper/2306.12757/) (2023)
- [The Rate-Distortion-Perception-Classification Tradeoff: Joint Source Coding and Modulation via Inverse-Domain GANs](https://mlsanity.com/paper/2312.14792/) (2023)
- [Constrained Gaussian Wasserstein Optimal Transport with Commutative Covariance Matrices](https://mlsanity.com/paper/2503.03744/) (2025)
- [Training-Free Rate-Distortion-Perception Traversal With Diffusion](https://mlsanity.com/paper/2603.04005/) (2026)
- [FlowCodec: One-Step Flow Prior for Generative Image Compression](https://mlsanity.com/paper/2606.21030/) (2026)
- [FLM: Frequency-Aware Language Models for Generative Image Compression](https://mlsanity.com/paper/2608.28687/) (2026)

## Abstract (arXiv)

Lossy compression algorithms are typically designed to achieve the lowest possible distortion at a given bit rate. However, recent studies show that pursuing high perceptual quality would lead to increase of the lowest achievable distortion (e.g., MSE). This paper provides nontrivial results theoretically revealing that, textit1) the cost of achieving perfect perception quality is exactly a doubling of the lowest achievable MSE distortion, textit2) an optimal encoder for the "classic" rate-distortion problem is also optimal for the perceptual compression problem, textit3) distortion loss is unnecessary for training a perceptual decoder. Further, we propose a novel training framework to achieve the lowest MSE distortion under perfect perception constraint at a given bit rate. This framework uses a GAN with discriminator conditioned on an MSE-optimized encoder, which is superior over the traditional framework using distortion plus adversarial loss. Experiments are provided to verify the theoretical finding and demonstrate the superiority of the proposed training framework.