# Multimodal LLM Integrated Semantic Communications for 6G Immersive Experiences

arXiv: https://arxiv.org/abs/2507.04621 · HTML: https://mlsanity.com/paper/2507.04621/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 7 Jul 2025.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2212.09506/) (2022)
- [CDDM: Channel Denoising Diffusion Models for Wireless Semantic Communications](https://mlsanity.com/paper/2309.08895/) (2023)
- [Generative Joint Source-Channel Coding for Semantic Image Transmission](https://mlsanity.com/paper/2211.13772/) (2022)
- [Deep Joint Source-Channel Coding for Semantic Communications](https://mlsanity.com/paper/2211.08747/) (2022)
- [Large AI Model Empowered Multimodal Semantic Communications](https://mlsanity.com/paper/2309.01249/) (2023)
- [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023)
- [Diffusion-Driven Semantic Communication for Generative Models with Bandwidth Constraints](https://mlsanity.com/paper/2407.18468/) (2024)
- [Toward Immersive Communications in 6G](https://mlsanity.com/paper/2303.05283/) (2023)
- [Digital-SC: Digital Semantic Communication with Adaptive Network Split and Learned Non-Linear Quantization](https://mlsanity.com/paper/2305.13553/) (2023)
- [Semantic Importance-Aware Communications with Semantic Correction Using Large Language Models](https://mlsanity.com/paper/2405.16011/) (2024)
- [Semantic Communication based on Large Language Model for Underwater Image Transmission](https://mlsanity.com/paper/2408.12616/) (2024)

## Built on by (most cited first, 3 of 3)

- [Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks](https://mlsanity.com/paper/2509.21259/) (2025)
- [TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications](https://mlsanity.com/paper/2603.00482/) (2026)
- [6G Needs Agents: Toward Agentic AI-Native Networks for Autonomous Intelligence](https://mlsanity.com/paper/2605.01546/) (2026)

## Abstract (arXiv)

6G networks promise revolutionary immersive communication experiences including augmented reality (AR), virtual reality (VR), and holographic communications. These applications demand high-dimensional multimodal data transmission and intelligent data processing in real-time, which is extremely challenging over resource-limited wireless communication systems. Moreover, a joint understanding of the environment, context, and user intent is essential to deliver task-relevant content effectively. This article presents a novel multimodal large language model (MLLM) integrated semantic communications framework, termed MLLM-SC, which fully leverages reasoning and generative capabilities of pre-trained foundation models for context-aware and task-oriented wireless communication. The MLLM-SC framework adopts a device-edge collaborative architecture. At the edge, MLLM-empowered semantic guidance module analyzes multimodal inputs, user intents, and channel conditions to generate importance-aware attention maps prioritizing semantically critical information. An importance-aware semantic encoder and a resource-adaptive semantic decoder are jointly designed and optimized, which can utilize the semantic guidance for adaptive bandwidth allocation and high-quality content reconstruction or generation. Extensive case studies on visual question answering for AR/VR applications and diffusion-driven image generation validate the effectiveness of MLLM-SC.