# Large Language Model-Driven Distributed Integrated Multimodal Sensing and Semantic Communications

arXiv: https://arxiv.org/abs/2505.18194 · HTML: https://mlsanity.com/paper/2505.18194/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 16 of 16)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Semantic Communication Systems for Speech Transmission](https://mlsanity.com/paper/2102.12605/) (2021)
- [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023)
- [When Large Language Model Agents Meet 6G Networks: Perception, Grounding, and Alignment](https://mlsanity.com/paper/2401.07764/) (2024)
- [Large AI Model Empowered Multimodal Semantic Communications](https://mlsanity.com/paper/2309.01249/) (2023)
- [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023)
- [Semantic Communication with Memory](https://mlsanity.com/paper/2303.12335/) (2023)
- [Large Generative Model Assisted 3D Semantic Communication](https://mlsanity.com/paper/2403.05783/) (2024)
- [Deep Multimodal Learning: Merging Sensory Data for Massive MIMO Channel Prediction](https://mlsanity.com/paper/2007.09366/) (2020)
- [SIMAC: A Semantic-Driven Integrated Multimodal Sensing And Communication Framework](https://mlsanity.com/paper/2503.08726/) (2025)
- [Semantic Communications with Computer Vision Sensing for Edge Video Transmission](https://mlsanity.com/paper/2503.07252/) (2025)
- [Role of Sensing and Computer Vision in 6G Wireless Communications](https://mlsanity.com/paper/2405.03945/) (2024)
- [FuseGrasp: Radar-Camera Fusion for Robotic Grasping of Transparent Objects](https://mlsanity.com/paper/2502.20037/) (2025)

## Built on by (most cited first, 2 of 2)

- [Semantic Edge-Cloud Communication for Real-Time Urban Traffic Surveillance with ViT and LLMs over Mobile Networks](https://mlsanity.com/paper/2509.21259/) (2025)
- [A Camera-Cooperative ISAC Framework for Multimodal Non-Cooperative UAVs Sensing](https://mlsanity.com/paper/2605.22090/) (2026)

## Abstract (arXiv)

Traditional single-modal sensing systems-based solely on either radio frequency (RF) or visual data-struggle to cope with the demands of complex and dynamic environments. Furthermore, single-device systems are constrained by limited perspectives and insufficient spatial coverage, which impairs their effectiveness in urban or non-line-of-sight scenarios. To overcome these challenges, we propose a novel large language model (LLM)-driven distributed integrated multimodal sensing and semantic communication (LLM-DiSAC) framework. Specifically, our system consists of multiple collaborative sensing devices equipped with RF and camera modules, working together with an aggregation center to enhance sensing accuracy. First, on sensing devices, LLM-DiSAC develops an RF-vision fusion network (RVFN), which employs specialized feature extractors for RF and visual data, followed by a cross-attention module for effective multimodal integration. Second, a LLM-based semantic transmission network (LSTN) is proposed to enhance communication efficiency, where the LLM-based decoder leverages known channel parameters, such as transceiver distance and signal-to-noise ratio (SNR), to mitigate semantic distortion. Third, at the aggregation center, a transformer-based aggregation model (TRAM) with an adaptive aggregation attention mechanism is developed to fuse distributed features and enhance sensing accuracy. To preserve data privacy, a two-stage distributed learning strategy is introduced, allowing local model training at the device level and centralized aggregation model training using intermediate features. Finally, evaluations on a synthetic multi-view RF-visual dataset generated by the Genesis simulation engine show that LLM-DiSAC achieves a good performance.