mlsanityData to 14 Oct 2026

Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networks

arXiv 2305.16483 ↗First posted 25 May 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

This paper considers a class of reinforcement learning problems, which involve systems with two types of states: stochastic and pseudo-stochastic. In such systems, stochastic states follow a stochastic transition kernel while the transitions of pseudo-stochastic states are deterministic given the stochastic states/transitions. We refer to such systems as mixed systems, which are widely used in various applications, including manufacturing systems, communication networks, and queueing networks. We propose a sample efficient RL method that accelerates learning by generating augmented data samples. The proposed algorithm is data-driven and learns the policy from data samples from both real and augmented samples. This method significantly improves learning by reducing the sample complexity such that the dataset only needs to have sufficient coverage of the stochastic states. We analyze the sample complexity of the proposed method under Fitted Q Iteration (FQI) and demonstrate that the optimality gap decreases as tildeO(√1/n+√1/m), where n is the number of real samples and m is the number of augmented samples per real sample. It is important to note that without augmented samples, the optimality gap is tildeO(1) due to insufficient data coverage of the pseudo-stochastic states. Our experimental results on multiple queueing network applications confirm that the proposed method indeed significantly accelerates learning in both deep Q-learning and deep policy gradient.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 26 of its 44 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

26 of 44 references.

References · 26

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  4. 2016Group Equivariant Convolutional Networkscited by 1,165
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  7. 2020Reinforcement Learning with Augmented Datacited by 468
  8. 2019General E(2)-Equivariant Steerable CNNscited by 446
  9. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  10. 2016Steerable CNNscited by 305
  11. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  12. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
14 more
  1. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  2. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  3. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 142
  4. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 140
  5. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  6. 2018Count-Based Exploration with the Successor Representationcited by 112
  7. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
  8. 2022SO(2)-Equivariant Reinforcement Learningcited by 84
  9. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  10. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 43
  11. 2020Queueing Network Controls via Deep Reinforcement Learningcited by 41
  12. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  13. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 26
  14. 2020Group Equivariant Deep Reinforcement Learningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 0; 2026: 1.

9 papers.

  1. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  2. 2024Overcoming the Curse of Dimensionality in Reinforcement Learning Through Approximate Factorizationcited by 6
  3. 2024Performance of NPG in Countable State-Space Average-Cost RLcited by 5
  4. 2024Approximating G(t)/GI/1 queues with deep learningcited by 4
  5. 2024Differentiable Discrete Event Simulation for Queuing Network Controlcited by 4
  6. 2024QGym: Scalable Simulation and Benchmarking of Queuing Network Controllerscited by 3
  7. 2024Adversarial Network Optimization under Bandit Feedback: Maximizing Utility in Non-Stationary Multi-Hop Networkscited by 1
  8. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  9. 2026Heterogeneous Vertiport Selection Optimization for On-Demand Air Taxi Services: A Deep Reinforcement Learning Approachcited by 0