# Deep Reinforcement Learning for mmWave Initial Beam Alignment

arXiv: https://arxiv.org/abs/2302.08969 · HTML: https://mlsanity.com/paper/2302.08969/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Dealing with Sparse Rewards in Reinforcement Learning](https://mlsanity.com/paper/1910.09281/) (2019)
- [Fast mmwave Beam Alignment via Correlated Bandit Learning](https://mlsanity.com/paper/1909.03313/) (2019)
- [Active Learning and CSI Acquisition for mmWave Initial Alignment](https://mlsanity.com/paper/1812.07722/) (2018)
- [Active Sensing for Communications by Learning](https://mlsanity.com/paper/2112.04075/) (2021)
- [Deep Reinforcement Learning based Blind mmWave MIMO Beam Alignment](https://mlsanity.com/paper/2001.09251/) (2020)
- [Reinforcement Learning of Beam Codebooks in Millimeter Wave and Terahertz MIMO Systems](https://mlsanity.com/paper/2102.11392/) (2021)

## Abstract (arXiv)

We investigate the applicability of deep reinforcement learning algorithms to the adaptive initial access beam alignment problem for mmWave communications using the state-of-the-art proximal policy optimization algorithm as an example. In comparison to recent unsupervised learning based approaches developed to tackle this problem, deep reinforcement learning has the potential to address a new and wider range of applications, since, in principle, no (differentiable) model of the channel and/or the whole system is required for training, and only agent-environment interactions are necessary to learn an algorithm (be it online or using a recorded dataset). We show that, although the chosen off-the-shelf deep reinforcement learning agent fails to perform well when trained on realistic problem sizes, introducing action space shaping in the form of beamforming modules vastly improves the performance, without sacrificing much generalizability. Using this add-on, the agent is able to deliver competitive performance to various state-of-the-art methods on simulated environments, even under realistic problem sizes. This demonstrates that through well-directed modification, deep reinforcement learning may have a chance to compete with other approaches in this area, opening up many straightforward extensions to other/similar scenarios.