# Controlling Rate, Distortion, and Realism: Towards a Single Comprehensive Neural Image Compression Model

arXiv: https://arxiv.org/abs/2405.16817 · HTML: https://mlsanity.com/paper/2405.16817/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 50 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 30 of 37)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [The relativistic discriminator: a key element missing from standard GAN](https://mlsanity.com/paper/1807.00734/) (2018)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Learned Image Compression with Mixed Transformer-CNN Architectures](https://mlsanity.com/paper/2303.14978/) (2023)
- [Checkerboard Context Model for Efficient Learned Image Compression](https://mlsanity.com/paper/2103.15306/) (2021)
- [Lossy Image Compression with Conditional Diffusion Models](https://mlsanity.com/paper/2209.06950/) (2022)
- [The Devil Is in the Details: Window-based Attention for Image Compression](https://mlsanity.com/paper/2203.08450/) (2022)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [Variable Rate Deep Image Compression With a Conditional Autoencoder](https://mlsanity.com/paper/1909.04802/) (2019)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)
- [Lossy Compression with Gaussian Diffusion](https://mlsanity.com/paper/2206.08889/) (2022)
- [Asymmetric Gained Deep Image Compression With Continuous Rate Adaptation](https://mlsanity.com/paper/2003.02012/) (2020)
- [Multi-Realism Image Compression with a Conditional Generator](https://mlsanity.com/paper/2212.13824/) (2022)
- [Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform](https://mlsanity.com/paper/2108.09551/) (2021)
- [High-Fidelity Image Compression with Score-based Generative Models](https://mlsanity.com/paper/2305.18231/) (2023)
- [Contextformer: A Transformer with Spatio-Channel Attention for Context Modeling in Learned Image Compression](https://mlsanity.com/paper/2203.02452/) (2022)

## Built on by (most cited first, 17 of 17)

- [Compressed Image Generation with Denoising Diffusion Codebook Models](https://mlsanity.com/paper/2502.01189/) (2025)
- [Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaptation](https://mlsanity.com/paper/2406.00758/) (2024)
- [All-in-One Image Coding for Joint Human-Machine Vision with Multi-Path Aggregation](https://mlsanity.com/paper/2409.19660/) (2024)
- [Rate-Distortion-Cognition Controllable Versatile Neural Image Compression](https://mlsanity.com/paper/2407.11700/) (2024)
- [Turbo-DDCM: Fast and Flexible Zero-Shot Diffusion-Based Image Compression](https://mlsanity.com/paper/2511.06424/) (2025)
- [Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior](https://mlsanity.com/paper/2511.22549/) (2025)
- [Conditional Hallucinations for Image Compression](https://mlsanity.com/paper/2410.19493/) (2024)
- [Motion Free B-frame Coding for Neural Video Compression](https://mlsanity.com/paper/2411.17160/) (2024)
- [Leveraging Diffusion Knowledge for Generative Image Compression with Fractal Frequency-Aware Band Learning](https://mlsanity.com/paper/2503.11321/) (2025)
- [Neural Weight Compression for Language Models](https://mlsanity.com/paper/2510.11234/) (2025)
- [Fast Training-free Perceptual Image Compression](https://mlsanity.com/paper/2506.16102/) (2025)
- [Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification](https://mlsanity.com/paper/2508.05489/) (2025)
- [Comp-X: On Defining an Interactive Learned Image Compression Paradigm With Expert-driven LLM Agent](https://mlsanity.com/paper/2508.15243/) (2025)
- [HyperVQ: Enabling Hyperprior Entropy Modeling for VQ-Based Generative Image Compression](https://mlsanity.com/paper/2512.07192/) (2025)
- [Adaptive Fused Prior Transfer for Controllable Generative Image Compression](https://mlsanity.com/paper/2605.16817/) (2026)
- [MoECodec: Image Compression for joint human and machine perception via Mixture-of-Experts](https://mlsanity.com/paper/2606.21033/) (2026)
- [H-DDCM: Hadamard Codebooks for Fast Generative Image and Video Compression](https://mlsanity.com/paper/2610.18125/) (2026)

## Abstract (arXiv)

In recent years, neural network-driven image compression (NIC) has gained significant attention. Some works adopt deep generative models such as GANs and diffusion models to enhance perceptual quality (realism). A critical obstacle of these generative NIC methods is that each model is optimized for a single bit rate. Consequently, multiple models are required to compress images to different bit rates, which is impractical for real-world applications. To tackle this issue, we propose a variable-rate generative NIC model. Specifically, we explore several discriminator designs tailored for the variable-rate approach and introduce a novel adversarial loss. Moreover, by incorporating the newly proposed multi-realism technique, our method allows the users to adjust the bit rate, distortion, and realism with a single model, achieving ultra-controllability. Unlike existing variable-rate generative NIC models, our method matches or surpasses the performance of state-of-the-art single-rate generative NIC models while covering a wide range of bit rates using just one model. Code will be available at https://github.com/iwa-shi/CRDR