# Computer Vision Aided Beam Tracking in A Real-World Millimeter Wave Deployment

arXiv: https://arxiv.org/abs/2111.14803 · HTML: https://mlsanity.com/paper/2111.14803/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2021.

## Built on by (most cited first, 11 of 11)

- [Multimodal Transformers for Wireless Communications: A Case Study in Beam Prediction](https://mlsanity.com/paper/2309.11811/) (2023)
- [BeamLLM: Vision-Empowered mmWave Beam Prediction with Large Language Models](https://mlsanity.com/paper/2503.10432/) (2025)
- [M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models](https://mlsanity.com/paper/2506.14532/) (2025)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Multi-modal Image and Radio Frequency Fusion for Optimizing Vehicle Positioning](https://mlsanity.com/paper/2410.19788/) (2024)
- [Multi-modal Data based Semi-Supervised Learning for Vehicle Positioning](https://mlsanity.com/paper/2410.20680/) (2024)
- [HawkRover: An Autonomous mmWave Vehicular Communication Testbed with Multi-sensor Fusion and Deep Learning](https://mlsanity.com/paper/2401.01822/) (2024)
- [GPS-Aided Deep Learning for Beam Prediction and Tracking in UAV mmWave Communication](https://mlsanity.com/paper/2505.17530/) (2025)
- [AI-Native Integrated Sensing and Communications for Self-Organizing Wireless Networks: Architectures, Learning Paradigms, and System-Level Design](https://mlsanity.com/paper/2601.02398/) (2025)
- [Look Once, Beam Twice: Camera-Primed Real-Time Double-Directional mmWave Beam Management for Vehicular Connectivity](https://mlsanity.com/paper/2605.05071/) (2026)
- [Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning](https://mlsanity.com/paper/2606.22195/) (2026)

## Abstract (arXiv)

Millimeter-wave (mmWave) and terahertz (THz) communications require beamforming to acquire adequate receive signal-to-noise ratio (SNR). To find the optimal beam, current beam management solutions perform beam training over a large number of beams in pre-defined codebooks. The beam training overhead increases the access latency and can become infeasible for high-mobility applications. To reduce or even eliminate this beam training overhead, we propose to utilize the visual data, captured for example by cameras at the base stations, to guide the beam tracking/refining process. We propose a machine learning (ML) framework, based on an encoder-decoder architecture, that can predict the future beams using the previously obtained visual sensing information. Our proposed approach is evaluated on a large-scale real-world dataset, where it achieves an accuracy of 64.47% (and a normalized receive power of 97.66%) in predicting the future beam. This is achieved while requiring less than 1% of the beam training overhead of a corresponding baseline solution that uses a sequence of previous beams to predict the future one. This high performance and low overhead obtained on the real-world dataset demonstrate the potential of the proposed vision-aided beam tracking approach in real-world applications.