# Processing Network Controls via Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2205.02119 · HTML: https://mlsanity.com/paper/2205.02119/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 142 references that are in the map.
- First posted on arXiv 1 May 2022.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Applications of Deep Reinforcement Learning in Communications and Networking: A Survey](https://mlsanity.com/paper/1810.07862/) (2018)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [Reinforcement Learning for Molecular Design Guided by Quantum Mechanics](https://mlsanity.com/paper/2002.07717/) (2020)
- [A Deep Value-network Based Approach for Multi-Driver Order Dispatching](https://mlsanity.com/paper/2106.04493/) (2021)
- [Trust Region-Guided Proximal Policy Optimization](https://mlsanity.com/paper/1901.10314/) (2019)
- [Reinforcement Learning for Ridesharing: An Extended Survey](https://mlsanity.com/paper/2105.01099/) (2021)
- [On-Policy Deep Reinforcement Learning for the Average-Reward Criterion](https://mlsanity.com/paper/2106.07329/) (2021)
- [MOVI: A Model-Free Approach to Dynamic Fleet Management](https://mlsanity.com/paper/1804.04758/) (2018)
- [Linear Programming for Large-Scale Markov Decision Problems](https://mlsanity.com/paper/1402.6763/) (2014)
- [A Closer Look at Deep Policy Gradients](https://mlsanity.com/paper/1811.02553/) (2018)
- [Scalable Deep Reinforcement Learning for Ride-Hailing](https://mlsanity.com/paper/2009.14679/) (2020)
- [Optimizing Online Matching for Ride-Sourcing Services with Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1902.06228/) (2019)
- [Refined Policy Improvement Bounds for MDPs](https://mlsanity.com/paper/2107.08068/) (2021)

## Abstract (arXiv)

Novel advanced policy gradient (APG) algorithms, such as proximal policy optimization (PPO), trust region policy optimization, and their variations, have become the dominant reinforcement learning (RL) algorithms because of their ease of implementation and good practical performance. This dissertation is concerned with theoretical justification and practical application of the APG algorithms for solving processing network control optimization problems. Processing network control problems are typically formulated as Markov decision process (MDP) or semi-Markov decision process (SMDP) problems that have several unconventional for RL features: infinite state spaces, unbounded costs, long-run average cost objectives. Policy improvement bounds play a crucial role in the theoretical justification of the APG algorithms. In this thesis we refine existing bounds for MDPs with finite state spaces and prove novel policy improvement bounds for classes of MDPs and SMDPs used to model processing network operations. We consider two examples of processing network control problems and customize the PPO algorithm to solve them. First, we consider parallel-server and multiclass queueing networks controls. Second, we consider the drivers repositioning problem in a ride-hailing service system. For both examples the PPO algorithm with auxiliary modifications consistently generates control policies that outperform state-of-art heuristics.