# Reinforcement Learning of Beam Codebooks in Millimeter Wave and Terahertz MIMO Systems

arXiv: https://arxiv.org/abs/2102.11392 · HTML: https://mlsanity.com/paper/2102.11392/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2021.

## Built on by (most cited first, 5 of 5)

- [Camera Based mmWave Beam Prediction: Towards Multi-Candidate Real-World Scenarios](https://mlsanity.com/paper/2308.06868/) (2023)
- [Vehicle Cameras Guide mmWave Beams: Approach and Real-World V2V Demonstration](https://mlsanity.com/paper/2308.10362/) (2023)
- [Deep Reinforcement Learning for mmWave Initial Beam Alignment](https://mlsanity.com/paper/2302.08969/) (2023)
- [Unlocking Metasurface Practicality for B5G Networks: AI-assisted RIS Planning](https://mlsanity.com/paper/2310.10330/) (2023)
- [Causality-Driven Reinforcement Learning for Joint Communication and Sensing](https://mlsanity.com/paper/2409.15329/) (2024)

## Abstract (arXiv)

Millimeter wave (mmWave) and terahertz MIMO systems rely on pre-defined beamforming codebooks for both initial access and data transmission. Being pre-defined, however, these codebooks are commonly not optimized for specific environments, user distributions, and/or possible hardware impairments. This leads to large codebook sizes with high beam training overhead which increases the initial access/tracking latency and makes it hard for these systems to support highly mobile applications. To overcome these limitations, this paper develops a deep reinforcement learning framework that learns how to iteratively optimize the codebook beam patterns (shapes) relying only on the receive power measurements and without requiring any explicit channel knowledge. The developed model learns how to autonomously adapt the beam patterns to best match the surrounding environment, user distribution, hardware impairments, and array geometry. Further, this approach does not require any knowledge about the channel, array geometry, RF hardware, or user positions. To reduce the learning time, the proposed model designs a novel Wolpertinger-variant architecture that is capable of efficiently searching for an optimal policy in a large discrete action space, which is important for large antenna arrays with quantized phase shifters. This complex-valued neural network architecture design respects the practical RF hardware constraints such as the constant-modulus and quantized phase shifter constraints. Simulation results based on the publicly available DeepMIMO dataset confirm the ability of the developed framework to learn near-optimal beam patterns for both line-of-sight (LOS) and non-LOS scenarios and for arrays with hardware impairments without requiring any channel knowledge.