# Text + Sketch: Image Compression at Ultra Low Rates

arXiv: https://arxiv.org/abs/2307.01944 · HTML: https://mlsanity.com/paper/2307.01944/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 4 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [CogView: Mastering Text-to-Image Generation via Transformers](https://mlsanity.com/paper/2105.13290/) (2021)
- [Video Enhancement with Task-Oriented Flow](https://mlsanity.com/paper/1711.09078/) (2017)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff](https://mlsanity.com/paper/1901.07821/) (2019)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Lossy Image Compression with Conditional Diffusion Models](https://mlsanity.com/paper/2209.06950/) (2022)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Nonlinear Transform Coding](https://mlsanity.com/paper/2007.03034/) (2020)
- [Lossy Compression with Gaussian Diffusion](https://mlsanity.com/paper/2206.08889/) (2022)
- [Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models](https://mlsanity.com/paper/2211.07793/) (2022)
- [Towards improved lossy image compression: Human image reconstruction with public-domain images](https://mlsanity.com/paper/1810.11137/) (2018)

## Built on by (most cited first, 30 of 67)

- [Towards image compression with perfect realism at ultra-low bitrates](https://mlsanity.com/paper/2310.10325/) (2023)
- [Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior](https://mlsanity.com/paper/2404.18820/) (2024)
- [Lossy Image Compression with Foundation Diffusion Models](https://mlsanity.com/paper/2404.08580/) (2024)
- [StableCodec: Taming One-Step Diffusion for Extreme Image Compression](https://mlsanity.com/paper/2506.21977/) (2025)
- [MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model](https://mlsanity.com/paper/2402.16749/) (2024)
- [One-Step Diffusion-Based Image Compression with Semantic Distillation](https://mlsanity.com/paper/2505.16687/) (2025)
- [Lossy Compression with Pretrained Diffusion Models](https://mlsanity.com/paper/2501.09815/) (2025)
- [Generative Latent Coding for Ultra-Low Bitrate Image and Video Compression](https://mlsanity.com/paper/2505.16177/) (2025)
- [Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity](https://mlsanity.com/paper/2403.02944/) (2024)
- [RDEIC: Accelerating Diffusion-Based Extreme Image Compression with Relay Residual Diffusion](https://mlsanity.com/paper/2410.02640/) (2024)
- [Generative Latent Coding for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2512.20194/) (2025)
- [DLF: Extreme Image Compression with Dual-generative Latent Fusion](https://mlsanity.com/paper/2503.01428/) (2025)
- [Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion](https://mlsanity.com/paper/2505.08281/) (2025)
- [PerCo (SD): Open Perceptual Compression](https://mlsanity.com/paper/2409.20255/) (2024)
- [Generative Neural Video Compression via Video Diffusion Prior](https://mlsanity.com/paper/2512.05016/) (2025)
- [Semantics-Guided Diffusion for Deep Joint Source-Channel Coding in Wireless Image Transmission](https://mlsanity.com/paper/2501.01138/) (2025)
- [Diffusion-Aided Joint Source Channel Coding For High Realism Wireless Image Transmission](https://mlsanity.com/paper/2404.17736/) (2024)
- [PICD: Versatile Perceptual Image Compression with Diffusion Rendering](https://mlsanity.com/paper/2505.05853/) (2025)
- [CoD: A Diffusion Foundation Model for Image Compression](https://mlsanity.com/paper/2511.18706/) (2025)
- [Generative Video Semantic Communication via Multimodal Semantic Fusion with Large Model](https://mlsanity.com/paper/2502.13838/) (2025)
- [Bridging the Gap between Gaussian Diffusion Models and Universal Quantization for Image Compression](https://mlsanity.com/paper/2504.02579/) (2025)
- [Ultra-Low Bitrate Perceptual Image Compression with Shallow Encoder](https://mlsanity.com/paper/2512.12229/) (2025)
- [CoD-Lite: Real-Time Diffusion-Based Generative Image Compression](https://mlsanity.com/paper/2604.12525/) (2026)
- [Semantic Similarity Score for Measuring Visual Similarity at Semantic Level](https://mlsanity.com/paper/2406.03865/) (2024)
- [Extremely low-bitrate Image Compression Semantically Disentangled by LMMs from a Human Perception Perspective](https://mlsanity.com/paper/2503.00399/) (2025)
- [Dual-Representation Image Compression at Ultra-Low Bitrates via Explicit Semantics and Implicit Textures](https://mlsanity.com/paper/2602.05213/) (2026)
- [Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression](https://mlsanity.com/paper/2604.10546/) (2026)
- [Semantic Feature Decomposition based Semantic Communication System of Images with Large-scale Visual Generation Models](https://mlsanity.com/paper/2410.20126/) (2024)
- [LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression](https://mlsanity.com/paper/2411.13033/) (2024)
- [A Lightweight Model for Perceptual Image Compression via Implicit Priors](https://mlsanity.com/paper/2502.13988/) (2025)

## Abstract (arXiv)

Recent advances in text-to-image generative models provide the ability to generate high-quality images from short text descriptions. These foundation models, when pre-trained on billion-scale datasets, are effective for various downstream tasks with little or no further training. A natural question to ask is how such models may be adapted for image compression. We investigate several techniques in which the pre-trained models can be directly used to implement compression schemes targeting novel low rate regimes. We show how text descriptions can be used in conjunction with side information to generate high-fidelity reconstructions that preserve both semantics and spatial structure of the original. We demonstrate that at very low bit-rates, our method can significantly improve upon learned compressors in terms of perceptual and semantic fidelity, despite no end-to-end training.