# ViWi Vision-Aided mmWave Beam Tracking: Dataset, Task, and Baseline Solutions

arXiv: https://arxiv.org/abs/2002.02445 · HTML: https://mlsanity.com/paper/2002.02445/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2020.

## Built on by (most cited first, 9 of 9)

- [Vision-Aided 6G Wireless Communications: Blockage Prediction and Proactive Handoff](https://mlsanity.com/paper/2102.09527/) (2021)
- [Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network](https://mlsanity.com/paper/2201.04712/) (2022)
- [Camera Based mmWave Beam Prediction: Towards Multi-Candidate Real-World Scenarios](https://mlsanity.com/paper/2308.06868/) (2023)
- [Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey](https://mlsanity.com/paper/2601.03181/) (2026)
- [Vision-Aided Beam Tracking: Explore the Proper Use of Camera Images with Deep Learning](https://mlsanity.com/paper/2109.14686/) (2021)
- [Multi-modal Image and Radio Frequency Fusion for Optimizing Vehicle Positioning](https://mlsanity.com/paper/2410.19788/) (2024)
- [Multi-modal Data based Semi-Supervised Learning for Vehicle Positioning](https://mlsanity.com/paper/2410.20680/) (2024)
- [Vision Aided Channel Prediction for Vehicular Communications: A Case Study of Received Power Prediction Using RGB Images](https://mlsanity.com/paper/2501.18618/) (2025)
- [Deep Learning for THz Channel Estimation and Beamforming Prediction via Sub-6GHz Channel](https://mlsanity.com/paper/2411.15589/) (2024)

## Abstract (arXiv)

Vision-aided wireless communication is motivated by the recent advances in deep learning and computer vision as well as the increasing dependence on line-of-sight links in millimeter wave (mmWave) and terahertz systems. By leveraging vision, this new research direction enables an interesting set of new capabilities such as vision-aided mmWave beam and blockage prediction, proactive hand-off, and resource allocation among others. These capabilities have the potential of reliably supporting highly-mobile applications such as vehicular/drone communications and wireless virtual/augmented reality in mmWave and terahertz systems. Investigating these interesting applications, however, requires the development of special dataset and machine learning tasks. Based on the Vision-Wireless (ViWi) dataset generation framework [1], this paper develops an advanced and realistic scenario/dataset that features multiple base stations, mobile users, and rich dynamics. Enabled by this dataset, the paper defines the vision-wireless mmWave beam tracking task (ViWi-BT) and proposes a baseline solution that can provide an initial benchmark for the future ViWi-BT algorithms.