# Queueing Network Controls via Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2008.01644 · HTML: https://mlsanity.com/paper/2008.01644/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2020.

## Built on by (most cited first, 30 of 41)

- [ORLM: A Customizable Framework in Training Large Models for Automated Optimization Modeling](https://mlsanity.com/paper/2405.17743/) (2024)
- [Policy Optimization for Continuous Reinforcement Learning](https://mlsanity.com/paper/2305.18901/) (2023)
- [Hindsight Learning for MDPs with Exogenous Inputs](https://mlsanity.com/paper/2207.06272/) (2022)
- [Average-Reward Reinforcement Learning with Trust Region Methods](https://mlsanity.com/paper/2106.03442/) (2021)
- [Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation](https://mlsanity.com/paper/2401.13884/) (2024)
- [Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networks](https://mlsanity.com/paper/2305.16483/) (2023)
- [Learning and Information in Stochastic Networks and Queues](https://mlsanity.com/paper/2105.08769/) (2021)
- [Logarithmic regret bounds for continuous-time average-reward Markov decision processes](https://mlsanity.com/paper/2205.11168/) (2022)
- [Efficient Reinforcement Learning for Routing Jobs in Heterogeneous Queueing Systems](https://mlsanity.com/paper/2402.01147/) (2024)
- [Offline Estimation of Controlled Markov Chains: Minimaxity and Sample Complexity](https://mlsanity.com/paper/2211.07092/) (2022)
- [Performance of NPG in Countable State-Space Average-Cost RL](https://mlsanity.com/paper/2405.20467/) (2024)
- [Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agents](https://mlsanity.com/paper/2101.07312/) (2021)
- [Minimax Optimal Estimation of Stability Under Distribution Shift](https://mlsanity.com/paper/2212.06338/) (2022)
- [Refined Policy Improvement Bounds for MDPs](https://mlsanity.com/paper/2107.08068/) (2021)
- [Online Learning and Optimization for Queues with Unknown Demand Curve and Service Distribution](https://mlsanity.com/paper/2303.03399/) (2023)
- [Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Management](https://mlsanity.com/paper/2406.05358/) (2024)
- [Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2409.14557/) (2024)
- [QGym: Scalable Simulation and Benchmarking of Queuing Network Controllers](https://mlsanity.com/paper/2410.06170/) (2024)
- [Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space](https://mlsanity.com/paper/2306.02574/) (2023)
- [QSpark: Towards Reliable Qiskit Code Generation](https://mlsanity.com/paper/2507.12642/) (2025)
- [Implicit Updates for Average-Reward Temporal Difference Learning](https://mlsanity.com/paper/2510.06149/) (2025)
- [Finding Optimal Policy for Queueing Models: New Parameterization](https://mlsanity.com/paper/2206.10073/) (2022)
- [Optimal Control of Multiclass Fluid Queueing Networks: A Machine Learning Approach](https://mlsanity.com/paper/2307.12405/) (2023)
- [Convergence of Natural Policy Gradient for a Family of Infinite-State Queueing MDPs](https://mlsanity.com/paper/2402.05274/) (2024)
- [Intervention-Assisted Policy Gradient Methods for Online Stochastic Queuing Network Optimization: Technical Report](https://mlsanity.com/paper/2404.04106/) (2024)
- [Adversarial Network Optimization under Bandit Feedback: Maximizing Utility in Non-Stationary Multi-Hop Networks](https://mlsanity.com/paper/2408.16215/) (2024)
- [A Novel Switch-Type Policy Network for Resource Allocation Problems: Technical Report](https://mlsanity.com/paper/2501.11136/) (2025)
- [Can machines solve general queueing systems?](https://mlsanity.com/paper/2202.01729/) (2022)
- [Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces](https://mlsanity.com/paper/2306.01896/) (2023)
- [Semi-Gradient SARSA Routing with Theoretical Guarantee on Traffic Stability and Weight Convergence](https://mlsanity.com/paper/2503.14927/) (2025)

## Abstract (arXiv)

Novel advanced policy gradient (APG) methods, such as Trust Region policy optimization and Proximal policy optimization (PPO), have become the dominant reinforcement learning algorithms because of their ease of implementation and good practical performance. A conventional setup for notoriously difficult queueing network control problems is a Markov decision problem (MDP) that has three features: infinite state space, unbounded costs, and long-run average cost objective. We extend the theoretical framework of these APG methods for such MDP problems. The resulting PPO algorithm is tested on a parallel-server system and large-size multiclass queueing networks. The algorithm consistently generates control policies that outperform state-of-art heuristics in literature in a variety of load conditions from light to heavy traffic. These policies are demonstrated to be near-optimal when the optimal policy can be computed. A key to the successes of our PPO algorithm is the use of three variance reduction techniques in estimating the relative value function via sampling. First, we use a discounted relative value function as an approximation of the relative value function. Second, we propose regenerative simulation to estimate the discounted relative value function. Finally, we incorporate the approximating martingale-process method into the regenerative estimator.