# Multi-Modality Deep Network for Extreme Learned Image Compression

arXiv: https://arxiv.org/abs/2304.13583 · HTML: https://mlsanity.com/paper/2304.13583/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 52 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [Residual Non-local Attention Networks for Image Restoration](https://mlsanity.com/paper/1903.10082/) (2019)
- [Deep Contextual Video Compression](https://mlsanity.com/paper/2109.15047/) (2021)
- [Generative Adversarial Networks for Extreme Learned Image Compression](https://mlsanity.com/paper/1804.02958/) (2018)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Variable Rate Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06085/) (2015)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Context-adaptive Entropy Model for End-to-end Optimized Image Compression](https://mlsanity.com/paper/1809.10452/) (2018)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [Real-Time Adaptive Image Compression](https://mlsanity.com/paper/1705.05823/) (2017)
- [Enhanced Invertible Encoding for Learned Image Compression](https://mlsanity.com/paper/2108.03690/) (2021)
- [Non-local Attention Optimized Deep Image Compression](https://mlsanity.com/paper/1904.09757/) (2019)

## Built on by (most cited first, 16 of 16)

- [Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior](https://mlsanity.com/paper/2404.18820/) (2024)
- [Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity](https://mlsanity.com/paper/2403.02944/) (2024)
- [RDEIC: Accelerating Diffusion-Based Extreme Image Compression with Relay Residual Diffusion](https://mlsanity.com/paper/2410.02640/) (2024)
- [Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion](https://mlsanity.com/paper/2505.08281/) (2025)
- [Video Quality Assessment: A Comprehensive Survey](https://mlsanity.com/paper/2412.04508/) (2024)
- [A Lightweight Model for Perceptual Image Compression via Implicit Priors](https://mlsanity.com/paper/2502.13988/) (2025)
- [Map-Assisted Remote-Sensing Image Compression at Extremely Low Bitrates](https://mlsanity.com/paper/2409.01935/) (2024)
- [Perceptual Image Compression with Cooperative Cross-Modal Side Information](https://mlsanity.com/paper/2311.13847/) (2023)
- [Multi-Modality Deep Network for JPEG Artifacts Reduction](https://mlsanity.com/paper/2305.02760/) (2023)
- [Exploiting Inter-Image Similarity Prior for Low-Bitrate Remote Sensing Image Compression](https://mlsanity.com/paper/2407.12295/) (2024)
- [Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2606.01608/) (2026)
- [Addressing Imbalance for Class Incremental Learning in Medical Image Classification](https://mlsanity.com/paper/2407.13768/) (2024)
- [Joint Degradation-Aware Arbitrary-Scale Super-Resolution for Variable-Rate Extreme Image Compression](https://mlsanity.com/paper/2603.17408/) (2026)
- [Distributed Image Compression with Multimodal Side Information at Extremely Low Bitrates](https://mlsanity.com/paper/2605.22061/) (2026)
- [ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression](https://mlsanity.com/paper/2607.28020/) (2026)
- [Event-guided Neural Video Compression](https://mlsanity.com/paper/2610.02265/) (2026)

## Abstract (arXiv)

Image-based single-modality compression learning approaches have demonstrated exceptionally powerful encoding and decoding capabilities in the past few years , but suffer from blur and severe semantics loss at extremely low bitrates. To address this issue, we propose a multimodal machine learning method for text-guided image compression, in which the semantic information of text is used as prior information to guide image compression for better compression performance. We fully study the role of text description in different components of the codec, and demonstrate its effectiveness. In addition, we adopt the image-text attention module and image-request complement module to better fuse image and text features, and propose an improved multimodal semantic-consistent loss to produce semantically complete reconstructions. Extensive experiments, including a user study, prove that our method can obtain visually pleasing results at extremely low bitrates, and achieves a comparable or even better performance than state-of-the-art methods, even though these methods are at 2x to 4x bitrates of ours.