# Deep Reinforcement Learning for Uplink Scheduling in NOMA-URLLC Networks

arXiv: https://arxiv.org/abs/2308.14523 · HTML: https://mlsanity.com/paper/2308.14523/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 63 references that are in the map.
- First posted on arXiv 28 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [Approximate information state for approximate planning and reinforcement learning in partially observed systems](https://mlsanity.com/paper/2010.08843/) (2020)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Wireless Access in Ultra-Reliable Low-Latency Communication (URLLC)](https://mlsanity.com/paper/1810.06938/) (2018)
- [Grant-free Non-orthogonal Multiple Access for IoT: A Survey](https://mlsanity.com/paper/1910.06529/) (2019)
- [Distributive Dynamic Spectrum Access through Deep Reinforcement Learning: A Reservoir Computing Based Approach](https://mlsanity.com/paper/1810.11758/) (2018)
- [Joint Power and Blocklength Optimization for URLLC in a Factory Automation Scenario](https://mlsanity.com/paper/1911.13050/) (2019)
- [Cooperative Multi-Agent Reinforcement Learning Based Distributed Dynamic Spectrum Access in Cognitive Radio Networks](https://mlsanity.com/paper/2106.09274/) (2021)
- [Analyzing Grant-Free Access for URLLC Service](https://mlsanity.com/paper/2002.07842/) (2020)
- [Bayesian Transfer Reinforcement Learning with Prior Knowledge Rules](https://mlsanity.com/paper/1810.00468/) (2018)
- [Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications](https://mlsanity.com/paper/2002.08743/) (2020)
- [Joint Subcarrier and Power Allocation in NOMA: Optimal and Approximate Algorithms](https://mlsanity.com/paper/1910.00510/) (2019)
- [A General Deep Reinforcement Learning Framework for Grant-Free NOMA Optimization in mURLLC](https://mlsanity.com/paper/2101.00515/) (2021)

## Abstract (arXiv)

This article addresses the problem of Ultra Reliable Low Latency Communications (URLLC) in wireless networks, a framework with particularly stringent constraints imposed by many Internet of Things (IoT) applications from diverse sectors. We propose a novel Deep Reinforcement Learning (DRL) scheduling algorithm, named NOMA-PPO, to solve the Non-Orthogonal Multiple Access (NOMA) uplink URLLC scheduling problem involving strict deadlines. The challenge of addressing uplink URLLC requirements in NOMA systems is related to the combinatorial complexity of the action space due to the possibility to schedule multiple devices, and to the partial observability constraint that we impose to our algorithm in order to meet the IoT communication constraints and be scalable. Our approach involves 1) formulating the NOMA-URLLC problem as a Partially Observable Markov Decision Process (POMDP) and the introduction of an agent state, serving as a sufficient statistic of past observations and actions, enabling a transformation of the POMDP into a Markov Decision Process (MDP); 2) adapting the Proximal Policy Optimization (PPO) algorithm to handle the combinatorial action space; 3) incorporating prior knowledge into the learning agent with the introduction of a Bayesian policy. Numerical results reveal that not only does our approach outperform traditional multiple access protocols and DRL benchmarks on 3GPP scenarios, but also proves to be robust under various channel and traffic configurations, efficiently exploiting inherent time correlations.