# Self-Organizing mmWave MIMO Cell-Free Networks With Hybrid Beamforming: A Hierarchical DRL-Based Design

arXiv: https://arxiv.org/abs/2103.11823 · HTML: https://mlsanity.com/paper/2103.11823/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 41 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 10 of 10)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Soft Actor-Critic for Discrete Action Settings](https://mlsanity.com/paper/1910.07207/) (2019)
- [Scalable Cell-Free Massive MIMO Systems](https://mlsanity.com/paper/1908.03119/) (2019)
- [A Deep Q-Learning Method for Downlink Power Allocation in Multi-Cell Networks](https://mlsanity.com/paper/1904.13032/) (2019)
- [RF Chain Reduction for MIMO Systems: A Hardware Prototype](https://mlsanity.com/paper/1905.05315/) (2019)
- [Energy-Efficient Power Control in Cell-Free and User-Centric Massive MIMO at Millimeter Wave](https://mlsanity.com/paper/1903.11365/) (2019)
- [Downlink Training in Cell-Free Massive MIMO: A Blessing in Disguise](https://mlsanity.com/paper/1903.10046/) (2019)

## Built on by (most cited first, 3 of 3)

- [Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Models](https://mlsanity.com/paper/2502.07211/) (2025)
- [A Hierarchical DRL Approach for Resource Optimization in Multi-RIS Multi-Operator Networks](https://mlsanity.com/paper/2410.12320/) (2024)
- [Koopman-Based Generalization of Deep Reinforcement Learning With Application to Wireless Communications](https://mlsanity.com/paper/2503.02961/) (2025)

## Abstract (arXiv)

In a cell-free wireless network, distributed access points (APs) jointly serve all user equipments (UEs) within the their coverage area by using the same time/frequency resources. In this paper, we develop a novel downlink cell-free multiple-input multiple-output (MIMO) millimeter wave (mmWave) network architecture that enables all APs and UEs to dynamically self-partition into a set of independent cell-free subnetworks in a time-slot basis. For this, we propose several network partitioning algorithms based on deep reinforcement learning (DRL). Furthermore, to mitigate interference between different cell-free subnetworks, we develop a novel hybrid analog beamsteering-digital beamforming model that zero-forces interference among cell-free subnetworks and at the same time maximizes the instantaneous sum-rate of all UEs within each subnetwork. Specifically, the hybrid beamforming model is implemented by using a novel mixed DRL-convex optimization method in which analog beamsteering between APs and UEs is conducted based on DRL while digital beamforming is modeled and solved as a convex optimization problem. The DRL models for network clustering and hybrid beamsteering are combined into a single hierarchical DRL design that enables exchange of DRL agents' experiences during both network training and operation. We also benchmark the performance of DRL models for clustering and beamsteering in terms of network performance, convergence rate, and computational complexity.