# Multi-Modal Semantic Communication

arXiv: https://arxiv.org/abs/2512.15691 · HTML: https://mlsanity.com/paper/2512.15691/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 12 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 10 of 10)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Mask2Former for Video Instance Segmentation](https://mlsanity.com/paper/2112.10764/) (2021)
- [Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2408.00744/) (2024)
- [Task-Oriented Communications for NextG: End-to-End Deep Learning and AI Security Aspects](https://mlsanity.com/paper/2212.09668/) (2022)
- [Vulnerabilities of Deep Learning-Driven Semantic Communications to Backdoor (Trojan) Attacks](https://mlsanity.com/paper/2212.11205/) (2022)
- [Transformer-Aided Semantic Communications](https://mlsanity.com/paper/2405.01521/) (2024)
- [Efficient Semantic Communication Through Transformer-Aided Compression](https://mlsanity.com/paper/2412.01817/) (2024)

## Built on by (most cited first, 1 of 1)

- [Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission](https://mlsanity.com/paper/2609.16279/) (2026)

## Abstract (arXiv)

Semantic communication aims to transmit information most relevant to a task rather than raw data, offering significant gains in communication efficiency for applications such as telepresence, augmented reality, and remote sensing. Recent transformer-based approaches have used self-attention maps to identify informative regions within images, but they often struggle in complex scenes with multiple objects, where self-attention lacks explicit task guidance. To address this, we propose a novel Multi-Modal Semantic Communication framework that integrates text-based user queries to guide the information extraction process. Our proposed system employs a cross-modal attention mechanism that fuses visual features with language embeddings to produce soft relevance scores over the visual data. Based on these scores and the instantaneous channel bandwidth, we use an algorithm to transmit image patches at adaptive resolutions using independently trained encoder-decoder pairs, with total bitrate matching the channel capacity. At the receiver, the patches are reconstructed and combined to preserve task-critical information. This flexible and goal-driven design enables efficient semantic communication in complex and bandwidth-constrained environments.