# Cross Modal Compression: Towards Human-comprehensible Semantic Compression

arXiv: https://arxiv.org/abs/2209.02574 · HTML: https://mlsanity.com/paper/2209.02574/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 51 references that are in the map.
- First posted on arXiv 6 Sep 2022.

## Builds on (most cited first, 20 of 20)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [A Comprehensive Survey of Deep Learning for Image Captioning](https://mlsanity.com/paper/1810.04020/) (2018)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [To Create What You Tell: Generating Videos from Captions](https://mlsanity.com/paper/1804.08264/) (2018)
- [Jointly Modeling Embedding and Translation to Bridge Video and Language](https://mlsanity.com/paper/1505.01861/) (2015)
- [Deep feature compression for collaborative object detection](https://mlsanity.com/paper/1802.03931/) (2018)
- [Deep Reinforcement Learning-based Image Captioning with Embedding Reward](https://mlsanity.com/paper/1704.03899/) (2017)
- [Image Captioning with Deep Bidirectional LSTMs](https://mlsanity.com/paper/1604.00790/) (2016)
- [Learning based Facial Image Compression with Semantic Fidelity Metric](https://mlsanity.com/paper/1812.10067/) (2018)

## Built on by (most cited first, 7 of 7)

- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding](https://mlsanity.com/paper/2408.08093/) (2024)
- [T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates](https://mlsanity.com/paper/2507.07633/) (2025)
- [CMC-Bench: Towards a New Paradigm of Visual Signal Compression](https://mlsanity.com/paper/2406.09356/) (2024)
- [Perceptual Image Compression with Cooperative Cross-Modal Side Information](https://mlsanity.com/paper/2311.13847/) (2023)
- [Hierarchical Semantic Compression for Consistent Image Semantic Restoration](https://mlsanity.com/paper/2502.16799/) (2025)
- [Your Demands Deserve More Bits: Referring Semantic Image Compression at Ultra-low Bitrate](https://mlsanity.com/paper/2506.00526/) (2025)

## Abstract (arXiv)

Traditional image/video compression aims to reduce the transmission/storage cost with signal fidelity as high as possible. However, with the increasing demand for machine analysis and semantic monitoring in recent years, semantic fidelity rather than signal fidelity is becoming another emerging concern in image/video compression. With the recent advances in cross modal translation and generation, in this paper, we propose the cross modal compression~(CMC), a semantic compression framework for visual data, to transform the high redundant visual data~(such as image, video, etc.) into a compact, human-comprehensible domain~(such as text, sketch, semantic map, attributions, etc.), while preserving the semantic. Specifically, we first formulate the CMC problem as a rate-distortion optimization problem. Secondly, we investigate the relationship with the traditional image/video compression and the recent feature compression frameworks, showing the difference between our CMC and these prior frameworks. Then we propose a novel paradigm for CMC to demonstrate its effectiveness. The qualitative and quantitative results show that our proposed CMC can achieve encouraging reconstructed results with an ultrahigh compression ratio, showing better compression performance than the widely used JPEG baseline.