# Machine Learning on Camera Images for Fast mmWave Beamforming

arXiv: https://arxiv.org/abs/2102.07337 · HTML: https://mlsanity.com/paper/2102.07337/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 17 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 4 of 4)

- [Millimeter Wave Cellular Wireless Networks: Potentials and Challenges](https://mlsanity.com/paper/1401.2560/) (2014)
- [Machine Learning for Reliable mmWave Systems: Blockage Prediction and Proactive Handoff](https://mlsanity.com/paper/1807.02723/) (2018)
- [Proactive Received Power Prediction Using Machine Learning and Depth Images for mmWave Networks](https://mlsanity.com/paper/1803.09698/) (2018)
- [Prediction and Communication Co-design for Ultra-Reliable and Low-Latency Communications](https://mlsanity.com/paper/1909.05787/) (2019)

## Built on by (most cited first, 4 of 4)

- [Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network](https://mlsanity.com/paper/2201.04712/) (2022)
- [Multiverse at the Edge: Interacting Real World and Digital Twins for Wireless Beamforming](https://mlsanity.com/paper/2305.10350/) (2023)
- [Vision-Based Autonomous MM-Wave Reflector Using ArUco-Driven Angle-of-Arrival Estimation](https://mlsanity.com/paper/2506.05195/) (2025)
- [Look Once, Beam Twice: Camera-Primed Real-Time Double-Directional mmWave Beam Management for Vehicular Connectivity](https://mlsanity.com/paper/2605.05071/) (2026)

## Abstract (arXiv)

Perfect alignment in chosen beam sectors at both transmit- and receive-nodes is required for beamforming in mmWave bands. Current 802.11ad WiFi and emerging 5G cellular standards spend up to several milliseconds exploring different sector combinations to identify the beam pair with the highest SNR. In this paper, we propose a machine learning (ML) approach with two sequential convolutional neural networks (CNN) that uses out-of-band information, in the form of camera images, to (i) rapidly identify the locations of the transmitter and receiver nodes, and then (ii) return the optimal beam pair. We experimentally validate this intriguing concept for indoor settings using the NI 60GHz mmwave transceiver. Our results reveal that our ML approach reduces beamforming related exploration time by 93% under different ambient lighting conditions, with an error of less than 1% compared to the time-intensive deterministic method defined by the current standards.