# Camera Based mmWave Beam Prediction: Towards Multi-Candidate Real-World Scenarios

arXiv: https://arxiv.org/abs/2308.06868 · HTML: https://mlsanity.com/paper/2308.06868/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Instance-aware Semantic Segmentation via Multi-task Network Cascades](https://mlsanity.com/paper/1512.04412/) (2015)
- [DeepSense 6G: A Large-Scale Real-World Multi-Modal Sensing and Communication Dataset](https://mlsanity.com/paper/2211.09769/) (2022)
- [Deep Learning for mmWave Beam and Blockage Prediction Using Sub-6GHz Channels](https://mlsanity.com/paper/1910.02900/) (2019)
- [Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff](https://mlsanity.com/paper/2102.09527/) (2021)
- [Millimeter Wave Base Stations with Cameras: Vision Aided Beam and Blockage Prediction](https://mlsanity.com/paper/1911.06255/) (2019)
- [Real-Time Digital Twins: Vision and Research Directions for 6G and Beyond](https://mlsanity.com/paper/2301.11283/) (2023)
- [What Will 5G Be?](https://mlsanity.com/paper/1405.2957/) (2014)
- [Vision-Position Multi-Modal Beam Prediction Using Real Millimeter Wave Datasets](https://mlsanity.com/paper/2111.07574/) (2021)
- [LiDAR Aided Future Beam Prediction in Real-World Millimeter Wave V2I Communications](https://mlsanity.com/paper/2203.05548/) (2022)
- [ViWi: A Deep Learning Dataset Framework for Vision-Aided Wireless Communications](https://mlsanity.com/paper/1911.06257/) (2019)
- [Deep Learning for TDD and FDD Massive MIMO: Mapping Channels in Space and Frequency](https://mlsanity.com/paper/1905.03761/) (2019)
- [Radar Aided 6G Beam Prediction: Deep Learning Algorithms and Real-World Demonstration](https://mlsanity.com/paper/2111.09676/) (2021)
- [Position Aided Beam Prediction in the Real World: How Useful GPS Locations Actually Are?](https://mlsanity.com/paper/2205.09054/) (2022)
- [Digital Twin Based Beam Prediction: Can we Train in the Digital World and Deploy in Reality?](https://mlsanity.com/paper/2301.07682/) (2023)
- [Integrated Sensing and Communication for 6G: Ten Key Machine Learning Roles](https://mlsanity.com/paper/2208.02157/) (2022)
- [ViWi Vision-Aided mmWave Beam Tracking: Dataset, Task, and Baseline Solutions](https://mlsanity.com/paper/2002.02445/) (2020)
- [Applying Deep-Learning-Based Computer Vision to Wireless Communications: Methodologies, Opportunities, and Challenges](https://mlsanity.com/paper/2006.05782/) (2020)
- [Towards Real-World 6G Drone Communication: Position and Camera Aided Beam Prediction](https://mlsanity.com/paper/2205.12187/) (2022)
- [Neural Networks Based Beam Codebooks: Learning mmWave Massive MIMO Beams that Adapt to Deployment and Hardware](https://mlsanity.com/paper/2006.14501/) (2020)
- [Reinforcement Learning of Beam Codebooks in Millimeter Wave and Terahertz MIMO Systems](https://mlsanity.com/paper/2102.11392/) (2021)
- [Vision-Aided Dynamic Blockage Prediction for 6G Wireless Communication Networks](https://mlsanity.com/paper/2006.09902/) (2020)
- [Vision-Aided Radio: User Identity Match in Radio and Video Domains Using Machine Learning](https://mlsanity.com/paper/2010.07219/) (2020)
- [Environment Semantic Aided Communication: A Real World Demonstration for Beam Prediction](https://mlsanity.com/paper/2302.06736/) (2023)

## Built on by (most cited first, 7 of 7)

- [VOMTC: Vision Objects for Millimeter and Terahertz Communications](https://mlsanity.com/paper/2409.09330/) (2024)
- [Multi-Modal Sensing Aided mmWave Beamforming for V2V Communications with Transformers](https://mlsanity.com/paper/2509.11112/) (2025)
- [QuaMoE-DRF: Proactive Beam and Rate Adaptation via Multimodal Dynamic Radio Map Forecasting in ISAC Networks](https://mlsanity.com/paper/2607.00974/) (2026)
- [Multi-Modal Sensing and Fusion in mmWave Beamforming for Connected Vehicles: A Transformer Based Framework](https://mlsanity.com/paper/2602.13606/) (2026)
- [Environment-Aware Channel Prediction for Vehicular Communications: A Multimodal Visual Feature Fusion Framework](https://mlsanity.com/paper/2604.02396/) (2026)
- [Look Once, Beam Twice: Camera-Primed Real-Time Double-Directional mmWave Beam Management for Vehicular Connectivity](https://mlsanity.com/paper/2605.05071/) (2026)
- [Link Both Views: Cross-Image Routing Attention for Dual-Endpoint V2I Beam-Pair Power Prediction](https://mlsanity.com/paper/2610.18195/) (2026)

## Abstract (arXiv)

Leveraging sensory information to aid the millimeter-wave (mmWave) and sub-terahertz (sub-THz) beam selection process is attracting increasing interest. This sensory data, captured for example by cameras at the basestations, has the potential of significantly reducing the beam sweeping overhead and enabling highly-mobile applications. The solutions developed so far, however, have mainly considered single-candidate scenarios, i.e., scenarios with a single candidate user in the visual scene, and were evaluated using synthetic datasets. To address these limitations, this paper extensively investigates the sensing-aided beam prediction problem in a real-world multi-object vehicle-to-infrastructure (V2I) scenario and presents a comprehensive machine learning-based framework. In particular, this paper proposes to utilize visual and positional data to predict the optimal beam indices as an alternative to the conventional beam sweeping approaches. For this, a novel user (transmitter) identification solution has been developed, a key step in realizing sensing-aided multi-candidate and multi-user beam prediction solutions. The proposed solutions are evaluated on the large-scale real-world DeepSense 6G dataset. Experimental results in realistic V2I communication scenarios indicate that the proposed solutions achieve close to 100% top-5 beam prediction accuracy for the scenarios with single-user and close to 95% top-5 beam prediction accuracy for multi-candidate scenarios. Furthermore, the proposed approach can identify the probable transmitting candidate with more than 93% accuracy across the different scenarios. This highlights a promising approach for nearly eliminating the beam training overhead in mmWave/THz communication systems.