# LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression

arXiv: https://arxiv.org/abs/2411.13033 · HTML: https://mlsanity.com/paper/2411.13033/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modules](https://mlsanity.com/paper/2001.01568/) (2020)
- [High-Fidelity Generative Image Compression](https://mlsanity.com/paper/2006.09965/) (2020)
- [DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior](https://mlsanity.com/paper/2308.15070/) (2023)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images](https://mlsanity.com/paper/2403.11703/) (2024)
- [Towards image compression with perfect realism at ultra-low bitrates](https://mlsanity.com/paper/2310.10325/) (2023)
- [Multi-Realism Image Compression with a Conditional Generator](https://mlsanity.com/paper/2212.13824/) (2022)
- [Text + Sketch: Image Compression at Ultra Low Rates](https://mlsanity.com/paper/2307.01944/) (2023)
- [High-Fidelity Image Compression with Score-based Generative Models](https://mlsanity.com/paper/2305.18231/) (2023)
- [LLMZip: Lossless Text Compression using Large Language Models](https://mlsanity.com/paper/2306.04050/) (2023)
- [PO-ELIC: Perception-Oriented Efficient Learned Image Coding](https://mlsanity.com/paper/2205.14501/) (2022)
- [MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model](https://mlsanity.com/paper/2402.16749/) (2024)
- [Fidelity-Controllable Extreme Image Compression with Generative Adversarial Networks](https://mlsanity.com/paper/2008.10314/) (2020)
- [Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization](https://mlsanity.com/paper/2405.14221/) (2024)

## Built on by (most cited first, 4 of 4)

- [Extremely low-bitrate Image Compression Semantically Disentangled by LMMs from a Human Perception Perspective](https://mlsanity.com/paper/2503.00399/) (2025)
- [Low-Bitrate Video Compression through Semantic-Conditioned Diffusion](https://mlsanity.com/paper/2512.00408/) (2025)
- [Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression](https://mlsanity.com/paper/2606.01608/) (2026)
- [What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters](https://mlsanity.com/paper/2604.10017/) (2026)

## Abstract (arXiv)

Supported by powerful generative models, low-bitrate learned image compression (LIC) models utilizing perceptual metrics have become feasible. Some of the most advanced models achieve high compression rates and superior perceptual quality by using image captions as sub-information. This paper demonstrates that using a large multi-modal model (LMM), it is possible to generate captions and compress them within a single model. We also propose a novel semantic-perceptual-oriented fine-tuning method applicable to any LIC network, resulting in a 41.58% improvement in LPIPS BD-rate compared to existing methods. Our implementation and pre-trained weights are available at https://github.com/tokkiwa/ImageTextCoding.