# Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models

arXiv: https://arxiv.org/abs/2509.05925 · HTML: https://mlsanity.com/paper/2509.05925/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 7 Sep 2025.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [Image Coding for Machines with Edge Information Learning Using Segment Anything](https://mlsanity.com/paper/2403.04173/) (2024)
- [Learning Product Codebooks using Vector Quantized Autoencoders for Image Retrieval](https://mlsanity.com/paper/1807.04629/) (2018)
- [DeepSIC: Deep Semantic Image Compression](https://mlsanity.com/paper/1801.09468/) (2018)
- [DiffCP: Ultra-Low Bit Collaborative Perception via Diffusion Model](https://mlsanity.com/paper/2409.19592/) (2024)
- [LotteryCodec: Searching the Implicit Representation in a Random Network for Low-Complexity Image Compression](https://mlsanity.com/paper/2507.01204/) (2025)
- [Zero-Shot Semantic Communication with Multimodal Foundation Models](https://mlsanity.com/paper/2502.18200/) (2025)

## Built on by (most cited first, 4 of 4)

- [Semantic Compression via Multimodal Representation Learning](https://mlsanity.com/paper/2509.24431/) (2025)
- [Real-Time Inference for Distributed Multimodal Systems under Communication Delay Uncertainty](https://mlsanity.com/paper/2511.16225/) (2025)
- [Prompt-Guided Prefiltering for VLM Image Compression](https://mlsanity.com/paper/2604.00314/) (2026)
- [Adaptive Transform Coding for Semantic Compression](https://mlsanity.com/paper/2604.26492/) (2026)

## Abstract (arXiv)

Recent deep learning-based methods for lossy image compression achieve competitive rate-distortion performance through extensive end-to-end training and advanced architectures. However, emerging applications increasingly prioritize semantic preservation over pixel-level reconstruction and demand robust performance across diverse data distributions and downstream tasks. These challenges call for advanced semantic compression paradigms. Motivated by the zero-shot and representational capabilities of multimodal foundation models, we propose a novel semantic compression method based on the contrastive language-image pretraining (CLIP) model. Rather than compressing images for reconstruction, we propose compressing the CLIP feature embeddings into minimal bits while preserving semantic information across different tasks. Experiments show that our method maintains semantic integrity across benchmark datasets, achieving an average bit rate of approximately 2-3* 10(-3) bits per pixel. This is less than 5% of the bitrate required by mainstream image compression approaches for comparable performance. Remarkably, even under extreme compression, the proposed approach exhibits zero-shot robustness across diverse data distributions and downstream tasks.