# Communication-Efficient Orchestrations for URLLC Service via Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2307.13415 · HTML: https://mlsanity.com/paper/2307.13415/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Builds on (most cited first, 7 of 7)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Experienced Deep Reinforcement Learning with Generative Adversarial Networks (GANs) for Model-Free Ultra Reliable Low Latency Communication](https://mlsanity.com/paper/1911.03264/) (2019)
- [Wireless Access in Ultra-Reliable Low-Latency Communication (URLLC)](https://mlsanity.com/paper/1810.06938/) (2018)
- [Risk-Aware Resource Allocation for URLLC: Challenges and Strategies with Machine Learning](https://mlsanity.com/paper/1901.04292/) (2018)
- [Joint Pilot and Payload Power Allocation for Massive-MIMO-enabled URLLC IIoT Networks](https://mlsanity.com/paper/1912.12438/) (2019)
- [BSAC-CoEx: Coexistence of URLLC and Distributed Learning Services via Device Selection](https://mlsanity.com/paper/2212.11805/) (2022)

## Abstract (arXiv)

Ultra-reliable low latency communications (URLLC) service is envisioned to enable use cases with strict reliability and latency requirements in 5G. One approach for enabling URLLC services is to leverage Reinforcement Learning (RL) to efficiently allocate wireless resources. However, with conventional RL methods, the decision variables (though being deployed at various network layers) are typically optimized in the same control loop, leading to significant practical limitations on the control loop's delay as well as excessive signaling and energy consumption. In this paper, we propose a multi-agent Hierarchical RL (HRL) framework that enables the implementation of multi-level policies with different control loop timescales. Agents with faster control loops are deployed closer to the base station, while the ones with slower control loops are at the edge or closer to the core network providing high-level guidelines for low-level actions. On a use case from the prior art, with our HRL framework, we optimized the maximum number of retransmissions and transmission power of industrial devices. Our extensive simulation results on the factory automation scenario show that the HRL framework achieves better performance as the baseline single-agent RL method, with significantly less overhead of signal transmissions and delay compared to the one-agent RL methods.