Anytime Safe Reinforcement Learning
On the map: Deep reinforcement learning methods → Safe and constrained RL
Abstract (arXiv)
This paper considers the problem of solving constrained reinforcement learning problems with anytime guarantees, meaning that the algorithmic solution returns a safe policy regardless of when it is terminated. Drawing inspiration from anytime constrained optimization, we introduce Reinforcement Learning-based Safe Gradient Flow (RL-SGF), an on-policy algorithm which employs estimates of the value functions and their respective gradients associated with the objective and safety constraints for the current policy, and updates the policy parameters by solving a convex quadratically constrained quadratic program. We show that if the estimates are computed with a sufficiently large number of episodes (for which we provide an explicit bound), safe policies are updated to safe policies with a probability higher than a prescribed tolerance. We also show that iterates asymptotically converge to a neighborhood of a KKT point, whose size can be arbitrarily reduced by refining the estimates of the value function and their gradients. We illustrate the performance of RL-SGF in a navigation example.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 25 references that are in the map. First posted on arXiv 23 Apr 2025.
Builds on
17 of 25 references.
Closest ancestors · 1
Same direction, newest first
Other references · 16
- 2021Convex optimizationcited by 2,515
- 2017Constrained Policy Optimizationcited by 983
- 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
- 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
- 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
- 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
- 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
- 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
- 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
- 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
- 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
4 more
- 2023Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processescited by 26
- 2022Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithmcited by 20
- 2021Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processescited by 17
- 2022Control Barrier Function Based Design of Gradient Flows for Constrained Nonlinear Programmingcited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints
- 2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimation
- 2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learning
- 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
- 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
- 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time
- 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
- 2026Sampling-Based Safe Reinforcement Learning