# Multi-Modal Sensing and Fusion in mmWave Beamforming for Connected Vehicles: A Transformer Based Framework

arXiv: https://arxiv.org/abs/2602.13606 · HTML: https://mlsanity.com/paper/2602.13606/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 14 Feb 2026.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [MaxViT: Multi-Axis Vision Transformer](https://mlsanity.com/paper/2204.01697/) (2022)
- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [A Survey of Beam Management for mmWave and THz Communications Towards 6G](https://mlsanity.com/paper/2308.02135/) (2023)
- [Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network](https://mlsanity.com/paper/2201.04712/) (2022)
- [Position Aided Beam Prediction in the Real World: How Useful GPS Locations Actually Are?](https://mlsanity.com/paper/2205.09054/) (2022)
- [AI/ML for Beam Management in 5G-Advanced: A Standardization Perspective](https://mlsanity.com/paper/2309.10575/) (2023)
- [Multi-Modal Transformer and Reinforcement Learning-based Beam Management](https://mlsanity.com/paper/2410.19859/) (2024)
- [Camera Based mmWave Beam Prediction: Towards Multi-Candidate Real-World Scenarios](https://mlsanity.com/paper/2308.06868/) (2023)
- [Advancing Ultra-Reliable 6G: Transformer and Semantic Localization Empowered Robust Beamforming in Millimeter-Wave Communications](https://mlsanity.com/paper/2406.02000/) (2024)
- [Robust Beamforming Design for Near-Field DMA-NOMA mmWave Communications With Imperfect Position Information](https://mlsanity.com/paper/2409.15894/) (2024)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [mmWave Enabled Connected Autonomous Vehicles: A Use Case with V2V Cooperative Perception](https://mlsanity.com/paper/2310.09700/) (2023)
- [DeepSense-V2V: A Vehicle-to-Vehicle Multi-Modal Sensing, Localization, and Communications Dataset](https://mlsanity.com/paper/2406.17908/) (2024)
- [Multi-Modal Intelligent Channel Modeling: A New Modeling Paradigm via Synesthesia of Machines](https://mlsanity.com/paper/2411.03711/) (2024)
- [Position Aware 60 GHz mmWave Beamforming for V2V Communications Utilizing Deep Learning](https://mlsanity.com/paper/2402.01259/) (2024)
- [Beam Selection in ISAC using Contextual Bandit with Multi-modal Transformer and Transfer Learning](https://mlsanity.com/paper/2503.08937/) (2025)
- [Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning](https://mlsanity.com/paper/2504.06173/) (2025)
- [Synesthesia of Machines (SoM)-Enhanced Wideband Multi-User CSI Learning With LiDAR Sensing](https://mlsanity.com/paper/2408.12252/) (2024)
- [Multi-Modal Sensing Aided mmWave Beamforming for V2V Communications with Transformers](https://mlsanity.com/paper/2509.11112/) (2025)

## Abstract (arXiv)

Millimeter wave (mmWave) communication, utilizing beamforming techniques to address the inherent path loss limitation, is considered as one of the key technologies to support ever increasing high throughput and low latency demands of connected vehicles. However, adopting standard defined beamforming approach in highly dynamic vehicular environments often incurs high beam training overheads and reduction in the available airtime for communications, which is mainly due to exchanging pilot signals and exhaustive beam measurements. To this end, we present a multi-modal sensing and fusion learning framework as a potential alternative solution to reduce such overheads. In this framework, we first extract the representative features from the sensing modalities by modality specific encoders, then, utilize multi-head cross-modal attention to learn dependencies and correlations between different modalities, and subsequently fuse the multimodal features to obtain predicted top-k beams so that the best line-of-sight links can be proactively established. To show the generalizability of the proposed framework, we perform a comprehensive experiment in four different vehicle-to-infrastructure (V2I) and vehicle-to-vehicle (V2V) scenarios from real world multimodal and 60 GHz mmWave wireless sensing data. The experiment reveals that the proposed framework (i) achieves up to 96.72% accuracy on predicting top-15 beams correctly, (ii) incurs roughly 0.77 dB average power loss, and (iii) improves the overall latency and beam searching space overheads by 86.81% and 76.56% respectively for top-15 beams compared to standard defined approach.