# Lossy Image Compression with Foundation Diffusion Models

arXiv: https://arxiv.org/abs/2404.08580 · HTML: https://mlsanity.com/paper/2404.08580/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion for compression and wireless, one of 14 directions in Diffusion models.
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Placed in

- Diffusion models / [Diffusion for compression and wireless](https://mlsanity.com/diffusion/diffusion-for-compression-and-wireless/)

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Simple diffusion: End-to-end diffusion for high resolution images](https://mlsanity.com/paper/2301.11093/) (2023)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channel Contextual Adaptive Coding](https://mlsanity.com/paper/2203.10886/) (2022)
- [Diffusion models as plug-and-play priors](https://mlsanity.com/paper/2206.09012/) (2022)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [Scalable Adaptive Computation for Iterative Generation](https://mlsanity.com/paper/2212.11972/) (2022)
- [Lossy Image Compression with Conditional Diffusion Models](https://mlsanity.com/paper/2209.06950/) (2022)
- [Entroformer: A Transformer-based Entropy Model for Learned Image Compression](https://mlsanity.com/paper/2202.05492/) (2022)
- [Towards image compression with perfect realism at ultra-low bitrates](https://mlsanity.com/paper/2310.10325/) (2023)
- [Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models](https://mlsanity.com/paper/2301.11189/) (2023)
- [Multi-Realism Image Compression with a Conditional Generator](https://mlsanity.com/paper/2212.13824/) (2022)
- [Text + Sketch: Image Compression at Ultra Low Rates](https://mlsanity.com/paper/2307.01944/) (2023)
- [High-Fidelity Image Compression with Score-based Generative Models](https://mlsanity.com/paper/2305.18231/) (2023)

## Built on by (most cited first, 30 of 36)

- [StableCodec: Taming One-Step Diffusion for Extreme Image Compression](https://mlsanity.com/paper/2506.21977/) (2025)
- [BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation](https://mlsanity.com/paper/2407.17952/) (2024)
- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [One-Step Diffusion-Based Image Compression with Semantic Distillation](https://mlsanity.com/paper/2505.16687/) (2025)
- [Lossless data compression by large models](https://mlsanity.com/paper/2407.07723/) (2024)
- [PerCo (SD): Open Perceptual Compression](https://mlsanity.com/paper/2409.20255/) (2024)
- [PICD: Versatile Perceptual Image Compression with Diffusion Rendering](https://mlsanity.com/paper/2505.05853/) (2025)
- [GIViC: Generative Implicit Video Compression](https://mlsanity.com/paper/2503.19604/) (2025)
- [PSC: Posterior Sampling-Based Compression](https://mlsanity.com/paper/2407.09896/) (2024)
- [Bridging the Gap between Gaussian Diffusion Models and Universal Quantization for Image Compression](https://mlsanity.com/paper/2504.02579/) (2025)
- [Ultra-Low Bitrate Perceptual Image Compression with Shallow Encoder](https://mlsanity.com/paper/2512.12229/) (2025)
- [Timestep-Aware Diffusion Model for Extreme Image Rescaling](https://mlsanity.com/paper/2408.09151/) (2024)
- [HDCompression: Hybrid-Diffusion Image Compression for Ultra-Low Bitrates](https://mlsanity.com/paper/2502.07160/) (2025)
- [DiT-IC: Aligned Diffusion Transformer for Efficient Image Compression](https://mlsanity.com/paper/2603.13162/) (2026)
- [GVCC: Zero-Shot Video Compression via Codebook-Driven Stochastic Rectified Flow](https://mlsanity.com/paper/2603.26571/) (2026)
- [High Frequency Matters: Uncertainty Guided Image Compression with Wavelet Diffusion](https://mlsanity.com/paper/2407.12538/) (2024)
- [Single-step Diffusion for Image Compression at Ultra-Low Bitrates](https://mlsanity.com/paper/2506.16572/) (2025)
- [Steering One-Step Diffusion Model with Fidelity-Rich Decoder for Fast Image Compression](https://mlsanity.com/paper/2508.04979/) (2025)
- [Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior](https://mlsanity.com/paper/2511.22549/) (2025)
- [Guaranteed Conditional Diffusion: 3D Block-based Models for Scientific Data Compression](https://mlsanity.com/paper/2502.12951/) (2025)
- [Next-Frame Decoding for Ultra-Low-Bitrate Image Compression with Video Diffusion Priors](https://mlsanity.com/paper/2603.15129/) (2026)
- [Generative Latent Diffusion for Efficient Spatiotemporal Data Reduction](https://mlsanity.com/paper/2507.02129/) (2025)
- [CADC: Content Adaptive Diffusion-Based Generative Image Compression](https://mlsanity.com/paper/2602.21591/) (2026)
- [Advances in Diffusion-Based Generative Compression](https://mlsanity.com/paper/2601.18932/) (2026)
- [TextBoost: Boosting Scene Text Fidelity in Ultra-low Bitrate Image Compression](https://mlsanity.com/paper/2603.04115/) (2026)
- [FaSDiff: Balancing Perception and Semantics in Face Compression via Stable Diffusion Priors](https://mlsanity.com/paper/2505.05870/) (2025)
- [Keep It Real: Challenges in Attacking Compression-Based Adversarial Purification](https://mlsanity.com/paper/2508.05489/) (2025)
- [Region-Adaptive Generative Compression with Spatially Varying Diffusion Models](https://mlsanity.com/paper/2604.01122/) (2026)
- [DiV-INR: Extreme Low-Bitrate Diffusion Video Compression with INR Conditioning](https://mlsanity.com/paper/2604.08329/) (2026)
- [Active Sampling for Ultra-Low-Bit-Rate Video Compression via Conditional Controlled Diffusion](https://mlsanity.com/paper/2605.02849/) (2026)

## Abstract (arXiv)

Incorporating diffusion models in the image compression domain has the potential to produce realistic and detailed reconstructions, especially at extremely low bitrates. Previous methods focus on using diffusion models as expressive decoders robust to quantization errors in the conditioning signals, yet achieving competitive results in this manner requires costly training of the diffusion model and long inference times due to the iterative generative process. In this work we formulate the removal of quantization error as a denoising task, using diffusion to recover lost information in the transmitted image latent. Our approach allows us to perform less than 10% of the full diffusion generative process and requires no architectural changes to the diffusion model, enabling the use of foundation models as a strong prior without additional fine tuning of the backbone. Our proposed codec outperforms previous methods in quantitative realism metrics, and we verify that our reconstructions are qualitatively preferred by end users, even when other methods use twice the bitrate.