# SIMAC: A Semantic-Driven Integrated Multimodal Sensing And Communication Framework

arXiv: https://arxiv.org/abs/2503.08726 · HTML: https://mlsanity.com/paper/2503.08726/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [YOLOv10: Real-Time End-to-End Object Detection](https://mlsanity.com/paper/2405.14458/) (2024)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [A Survey on Homomorphic Encryption Schemes: Theory and Implementation](https://mlsanity.com/paper/1704.03578/) (2017)
- [BiFormer: Vision Transformer with Bi-Level Routing Attention](https://mlsanity.com/paper/2303.08810/) (2023)
- [Shunted Self-Attention via Multi-Scale Token Aggregation](https://mlsanity.com/paper/2111.15193/) (2021)
- [Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys](https://mlsanity.com/paper/2303.17220/) (2023)
- [CRAFT: Camera-Radar 3D Object Detection with Spatio-Contextual Fusion Transformer](https://mlsanity.com/paper/2209.06535/) (2022)
- [Large Language Model Enhanced Multi-Agent Systems for 6G Communications](https://mlsanity.com/paper/2312.07850/) (2023)
- [Large AI Model Empowered Multimodal Semantic Communications](https://mlsanity.com/paper/2309.01249/) (2023)
- [Large AI Model-Based Semantic Communications](https://mlsanity.com/paper/2307.03492/) (2023)
- [Integrated Sensing and Communication with Massive MIMO: A Unified Tensor Approach for Channel and Target Parameter Estimation](https://mlsanity.com/paper/2401.01738/) (2024)
- [Large Generative Model Assisted 3D Semantic Communication](https://mlsanity.com/paper/2403.05783/) (2024)
- [Deep Multimodal Learning: Merging Sensory Data for Massive MIMO Channel Prediction](https://mlsanity.com/paper/2007.09366/) (2020)
- [Explainable Semantic Federated Learning Enabled Industrial Edge Network for Fire Surveillance](https://mlsanity.com/paper/2412.19979/) (2024)
- [Vision Guided MIMO Radar Beamforming for Enhanced Vital Signs Detection in Crowds](https://mlsanity.com/paper/2306.10515/) (2023)
- [Green Beamforming Design for Integrated Sensing and Communication Systems: A Practical Approach Using Beam-Matching Error Metrics](https://mlsanity.com/paper/2310.13993/) (2023)
- [Optimizing Placement and Power Allocation in Reconfigurable Intelligent Sensing Surfaces for Enhanced Sensing and Communication Performance](https://mlsanity.com/paper/2409.06188/) (2024)

## Built on by (most cited first, 6 of 6)

- [CATCH: A Modular Cross-domain Adaptive Template with Hook](https://mlsanity.com/paper/2510.26582/) (2025)
- [Large Language Model-Driven Distributed Integrated Multimodal Sensing and Semantic Communications](https://mlsanity.com/paper/2505.18194/) (2025)
- [A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applications](https://mlsanity.com/paper/2601.22202/) (2026)
- [A Camera-Cooperative ISAC Framework for Multimodal Non-Cooperative UAVs Sensing](https://mlsanity.com/paper/2605.22090/) (2026)
- [LLM-Empowered Multimodal Fusion Framework for Autonomous Driving: Semantic Enhancement and Channel-Adaptive Design](https://mlsanity.com/paper/2607.01772/) (2026)
- [Adaptive Pilot Selection for Unified Semantic Communication and Semantic Sensing in ISAC](https://mlsanity.com/paper/2609.30891/) (2026)

## Abstract (arXiv)

Traditional single-modality sensing faces limitations in accuracy and capability, and its decoupled implementation with communication systems increases latency in bandwidth-constrained environments. Additionally, single-task-oriented sensing systems fail to address users' diverse demands. To overcome these challenges, we propose a semantic-driven integrated multimodal sensing and communication (SIMAC) framework. This framework leverages a joint source-channel coding architecture to achieve simultaneous sensing decoding and transmission of sensing results. Specifically, SIMAC first introduces a multimodal semantic fusion (MSF) network, which employs two extractors to extract semantic information from radar signals and images, respectively. MSF then applies cross-attention mechanisms to fuse these unimodal features and generate multimodal semantic representations. Secondly, we present a large language model (LLM)-based semantic encoder (LSE), where relevant communication parameters and multimodal semantics are mapped into a unified latent space and input to the LLM, enabling channel-adaptive semantic encoding. Thirdly, a task-oriented sensing semantic decoder (SSD) is proposed, in which different decoded heads are designed according to the specific needs of tasks. Simultaneously, a multi-task learning strategy is introduced to train the SIMAC framework, achieving diverse sensing services. Finally, experimental simulations demonstrate that the proposed framework achieves diverse sensing services and higher accuracy.