# Vision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets

arXiv: https://arxiv.org/abs/2111.07574 · HTML: https://mlsanity.com/paper/2111.07574/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2021.

## Built on by (most cited first, 24 of 24)

- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [Multimodal Transformers for Wireless Communications: A Case Study in Beam Prediction](https://mlsanity.com/paper/2309.11811/) (2023)
- [Multi-Modal Transformer and Reinforcement Learning-based Beam Management](https://mlsanity.com/paper/2410.19859/) (2024)
- [Camera Based mmWave Beam Prediction: Towards Multi-Candidate Real-World Scenarios](https://mlsanity.com/paper/2308.06868/) (2023)
- [Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework](https://mlsanity.com/paper/2504.05187/) (2025)
- [DeepSense-V2V: A Vehicle-to-Vehicle Multi-Modal Sensing, Localization, and Communications Dataset](https://mlsanity.com/paper/2406.17908/) (2024)
- [Vehicle Cameras Guide mmWave Beams: Approach and Real-World V2V Demonstration](https://mlsanity.com/paper/2308.10362/) (2023)
- [Generative AI Empowered LiDAR Point Cloud Generation with Multimodal Transformer](https://mlsanity.com/paper/2406.18542/) (2024)
- [Beam Selection in ISAC using Contextual Bandit with Multi-modal Transformer and Transfer Learning](https://mlsanity.com/paper/2503.08937/) (2025)
- [Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning](https://mlsanity.com/paper/2504.06173/) (2025)
- [Non-contact Multimodal Indoor Human Monitoring Systems: A Survey](https://mlsanity.com/paper/2312.07601/) (2023)
- [Communication-Efficient and Robust Multi-Modal Federated Learning via Latent-Space Consensus](https://mlsanity.com/paper/2603.19067/) (2026)
- [Millimeter Wave Drones with Cameras: Computer Vision Aided Wireless Beam Prediction](https://mlsanity.com/paper/2211.07569/) (2022)
- [Vision Aided Channel Prediction for Vehicular Communications: A Case Study of Received Power Prediction Using RGB Images](https://mlsanity.com/paper/2501.18618/) (2025)
- [Equivariant Multi-agent Reinforcement Learning for Multimodal Vehicle-to-Infrastructure Systems](https://mlsanity.com/paper/2604.06914/) (2026)
- [GPS-Aided Deep Learning for Beam Prediction and Tracking in UAV mmWave Communication](https://mlsanity.com/paper/2505.17530/) (2025)
- [Empower Low-Altitude Economy: A Reliability-Aware Dynamic Weighting Allocation for Multi-modal UAV Beam Prediction](https://mlsanity.com/paper/2512.24324/) (2025)
- [Agentic AI for Embodied-enhanced Beam Prediction in Low-Altitude Economy Networks](https://mlsanity.com/paper/2603.11392/) (2026)
- [Structure-Aware Multimodal LLM Framework for Trustworthy Near-Field Beam Prediction](https://mlsanity.com/paper/2603.16143/) (2026)
- [CSI-tuples-based 3D Channel Fingerprints Construction Assisted by MultiModal Learning](https://mlsanity.com/paper/2603.25288/) (2026)
- [Look Once, Beam Twice: Camera-Primed Real-Time Double-Directional mmWave Beam Management for Vehicular Connectivity](https://mlsanity.com/paper/2605.05071/) (2026)
- [Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning](https://mlsanity.com/paper/2606.22195/) (2026)
- [CM-MAE: A Physics-Guided Cross-Modal Self-Supervised Learning Framework for Vision-Wireless Applications](https://mlsanity.com/paper/2608.15972/) (2026)
- [Vision-Wireless Fusion for Multi-User Localization: A Cross-Modal Transformer Approach](https://mlsanity.com/paper/2609.23372/) (2026)

## Abstract (arXiv)

Enabling highly-mobile millimeter wave (mmWave) and terahertz (THz) wireless communication applications requires overcoming the critical challenges associated with the large antenna arrays deployed at these systems. In particular, adjusting the narrow beams of these antenna arrays typically incurs high beam training overhead that scales with the number of antennas. To address these challenges, this paper proposes a multi-modal machine learning based approach that leverages positional and visual (camera) data collected from the wireless communication environment for fast beam prediction. The developed framework has been tested on a real-world vehicular dataset comprising practical GPS, camera, and mmWave beam training data. The results show the proposed approach achieves more than ≈ 75% top-1 beam prediction accuracy and close to 100% top-3 beam prediction accuracy in realistic communication scenarios.