# Structured Reinforcement Learning for Delay-Optimal Data Transmission in Dense mmWave Networks

arXiv: https://arxiv.org/abs/2404.16920 · HTML: https://mlsanity.com/paper/2404.16920/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 59 references that are in the map.
- First posted on arXiv 25 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 19 of 19)

- [Combinatorial Sleeping Bandits with Fairness Constraints](https://mlsanity.com/paper/1901.04891/) (2019)
- [Learning Unknown Markov Decision Processes: A Thompson Sampling Approach](https://mlsanity.com/paper/1709.04570/) (2017)
- [Whittle index based Q-learning for restless bandits with average reward](https://mlsanity.com/paper/2004.14427/) (2020)
- [Regret Bounds for Thompson Sampling in Episodic Restless Bandit Problems](https://mlsanity.com/paper/1905.12673/) (2019)
- [Restless-UCB, an Efficient and Low-complexity Algorithm for Online Restless Bandits](https://mlsanity.com/paper/2011.02664/) (2020)
- [What Will 5G Be?](https://mlsanity.com/paper/1405.2957/) (2014)
- [Q-Learning Lagrange Policies for Multi-Action Restless Bandits](https://mlsanity.com/paper/2106.12024/) (2021)
- [An Asymptotically Optimal Index Policy for Finite-Horizon Restless Bandits](https://mlsanity.com/paper/1707.00205/) (2017)
- [Restless Bandits with Many Arms: Beating the Central Limit Theorem](https://mlsanity.com/paper/2107.11911/) (2021)
- [End-to-End Simulation of 5G mmWave Networks](https://mlsanity.com/paper/1705.02882/) (2017)
- [On learning Whittle index policy for restless bandits with scalable regret](https://mlsanity.com/paper/2202.03463/) (2022)
- [Dynamic priority allocation via restless bandit marginal productivity indices](https://mlsanity.com/paper/2304.06115/) (2023)
- [User Association in 5G Networks: A Survey and an Outlook](https://mlsanity.com/paper/1509.00338/) (2015)
- [User Association in Dense mmWave Networks as Restless Bandits](https://mlsanity.com/paper/2107.09153/) (2021)
- [A Reinforcement Learning Approach for Scheduling in mmWave Networks](https://mlsanity.com/paper/2108.00548/) (2021)
- [Low-Latency Millimeter-Wave Communications: Traffic Dispersion or Network Densification?](https://mlsanity.com/paper/1709.08410/) (2017)
- [Whittle Indexability in Egalitarian Processor Sharing Systems](https://mlsanity.com/paper/1707.02440/) (2017)
- [Delay-Aware Scheduling over mmWave/Sub-6 Dual Interfaces: A Reinforcement Learning Approach](https://mlsanity.com/paper/2005.08668/) (2020)
- [A Q-Learning-based Approach for Distributed Beam Scheduling in mmWave Networks](https://mlsanity.com/paper/2110.08704/) (2021)

## Abstract (arXiv)

We study the data packet transmission problem (mmDPT) in dense cell-free millimeter wave (mmWave) networks, i.e., users sending data packet requests to access points (APs) via uplinks and APs transmitting requested data packets to users via downlinks. Our objective is to minimize the average delay in the system due to APs' limited service capacity and unreliable wireless channels between APs and users. This problem can be formulated as a restless multi-armed bandits problem with fairness constraint (RMAB-F). Since finding the optimal policy for RMAB-F is intractable, existing learning algorithms are computationally expensive and not suitable for practical dynamic dense mmWave networks. In this paper, we propose a structured reinforcement learning (RL) solution for mmDPT by exploiting the inherent structure encoded in RMAB-F. To achieve this, we first design a low-complexity and provably asymptotically optimal index policy for RMAB-F. Then, we leverage this structure information to develop a structured RL algorithm called mmDPT-TS, which provably achieves an tildeO(√T) Bayesian regret. More importantly, mmDPT-TS is computation-efficient and thus amenable to practical implementation, as it fully exploits the structure of index policy for making decisions. Extensive emulation based on data collected in realistic mmWave networks demonstrate significant gains of mmDPT-TS over existing approaches.