# MMSense: Adapting Vision-based Foundation Model for Multi-task Multi-modal Wireless Sensing

arXiv: https://arxiv.org/abs/2511.12305 · HTML: https://mlsanity.com/paper/2511.12305/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 17 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing](https://mlsanity.com/paper/2305.10345/) (2023)
- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [Deep Learning for mmWave Beam and Blockage Prediction Using Sub-6GHz Channels](https://mlsanity.com/paper/1910.02900/) (2019)
- [LLM4CP: Adapting Large Language Models for Channel Prediction](https://mlsanity.com/paper/2406.14440/) (2024)
- [Deep Transfer Learning Based Downlink Channel Prediction for FDD Massive MIMO Systems](https://mlsanity.com/paper/1912.12265/) (2019)
- [LLM4WM: Adapting LLM for Wireless Multi-Tasking](https://mlsanity.com/paper/2501.12983/) (2025)
- [A Deep Learning-Based Framework for Low Complexity Multi-User MIMO Precoding Design](https://mlsanity.com/paper/2207.03765/) (2022)
- [Deep UL2DL: Channel Knowledge Transfer from Uplink to Downlink](https://mlsanity.com/paper/1812.07518/) (2018)
- [ICWLM: A Multi-Task Wireless Large Model via In-Context Learning](https://mlsanity.com/paper/2507.18167/) (2025)

## Built on by (most cited first, 1 of 1)

- [CSI-JEPA: Towards Foundation Representations for Ubiquitous Sensing with Minimal Supervision](https://mlsanity.com/paper/2605.14171/) (2026)

## Abstract (arXiv)

Large AI models have been widely adopted in wireless communications for channel modeling, beamforming, and resource optimization. However, most existing efforts remain limited to single-modality inputs and channel-specific objec- tives, overlooking the broader potential of large foundation models for unified wireless sensing. To bridge this gap, we propose MMSense, a multi-modal, multi-task foundation model that jointly addresses channel-centric, environment-aware, and human-centered sensing. Our framework integrates image, radar, LiDAR, and textual data by transforming them into vision- compatible representations, enabling effective cross-modal align- ment within a unified feature space. A modality gating mecha- nism adaptively fuses these representations, while a vision-based large language model backbone enables unified feature align- ment and instruction-driven task adaptation. Furthermore, task- specific sequential attention and uncertainty-based loss weighting mechanisms enhance cross-task generalization. Experiments on real wireless scenario datasets show that our approach outper- forms both task-specific and large-model baselines, confirming its strong generalization across heterogeneous sensing tasks.