mlsanityData to 7 Oct 2026

Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces

arXiv 2306.01896 ↗First posted 2 Jun 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. This latter objective is called stability and is especially important when the state space is unbounded, such that the states can be arbitrarily far from each other and the agent can drift far away from the desired states. For example, in stochastic queuing networks, where queues of waiting jobs can grow without bound, the desired state is all-zero queue lengths. Here, a stable policy ensures queue lengths are finite while an optimal policy minimizes queue lengths. Since an optimal policy is also stable, one would expect that RL algorithms would implicitly give us stable policies. However, in this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies, i.e., policies that drift far away from the desired state. We attribute this instability to poor credit-assignment for destabilizing actions. We then introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space. We conduct an empirical study on various queueing networks and traffic signal control problems and find that our approach performs competitively against strong baselines with knowledge of the transition dynamics. Our code is available here: https://github.com/Badger-RL/STOP.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 67 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

31 of 67 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  5. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  6. 2018Safe Exploration in Continuous Action Spacescited by 237
  7. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 218
  8. 2020Neural Lyapunov Controlcited by 142
  9. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  10. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  11. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  12. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
19 more
  1. 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
  2. 2019Hindsight Credit Assignmentcited by 74
  3. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 55
  4. 2020Learning and Planning in Average-Reward Markov Decision Processescited by 54
  5. 2020Queueing Network Controls via Deep Reinforcement Learningcited by 40
  6. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  7. 2018Safely Learning to Control the Constrained Linear Quadratic Regulatorcited by 31
  8. 2021Average-Reward Off-Policy Policy Evaluation with Function Approximationcited by 30
  9. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
  10. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 25
  11. 2022You Only Live Once: Single-Life Reinforcement Learningcited by 23
  12. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  13. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  14. 2022Lyapunov Design for Robust and Efficient Robotic Reinforcement Learningcited by 20
  15. 2019Episodic Learning with Control Lyapunov Functions for Uncertain Robotic Systemscited by 19
  16. 2016Learning Unknown Service Rates in Queues: A Multi-Armed Bandit Approachcited by 18
  17. 2020RL-QN: A Reinforcement Learning Framework for Optimal Control of Queueing Systemscited by 11
  18. 2023Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networkscited by 9
  19. 2020Stable Reinforcement Learning with Unbounded State Spacecited by 8

Built on by

Not cited yet.