# Vision Aided Environment Semantics Extraction and Its Application in mmWave Beam Selection

arXiv: https://arxiv.org/abs/2301.08973 · HTML: https://mlsanity.com/paper/2301.08973/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2023.

## Built on by (most cited first, 3 of 3)

- [CAVIAR: Co-simulation of 6G Communications, 3D Scenarios and AI for Digital Twins](https://mlsanity.com/paper/2401.03310/) (2024)
- [Multi-Modality Sensing in mmWave Beamforming for Connected Vehicles Using Deep Learning](https://mlsanity.com/paper/2504.06173/) (2025)
- [Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning](https://mlsanity.com/paper/2606.22195/) (2026)

## Abstract (arXiv)

In this letter, we propose a novel mmWave beam selection method based on the environment semantics extracted from user-side camera images. Specifically, we first define the environment semantics as the spatial distribution of the scatterers that affect the wireless propagation channels and utilize the keypoint detection technique to extract them from the input images. Then, we design a deep neural network with the environment semantics as the input that can output the optimal beam pairs at the mobile station (MS) and the base station (BS). Compared with the existing beam selection approaches that directly use images as the input, the proposed semantic-based method can explicitly obtain the environmental features that account for the propagation of wireless signals, thus reducing the storage and computational burden. Simulation results show that the proposed method can precisely estimate the location of the scatterers and outperform the existing works based on computer vision or light detection and ranging (LIDAR).