mlsanityData to 14 Oct 2026

Off-Policy Reinforcement Learning with Anytime Safety Guarantees via Robust Safe Gradient Flow

arXiv 2510.01492 ↗First posted 1 Oct 2025eess.SY cs.SYcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

This paper considers the problem of solving constrained reinforcement learning (RL) problems with anytime guarantees, meaning that the algorithmic solution must yield a constraint-satisfying policy at every iteration of its evolution. Our design is based on a discretization of the Robust Safe Gradient Flow (RSGF), a continuous-time dynamics for anytime constrained optimization whose forward invariance and stability properties we formally characterize. The proposed strategy, termed RSGF-RL, is an off-policy algorithm which uses episodic data to estimate the value functions and their gradients and updates the policy parameters by solving a convex quadratically constrained quadratic program. Our technical analysis combines statistical analysis, the theory of stochastic approximation, and convex analysis to determine the number of episodes sufficient to ensure that safe policies are updated to safe policies and to recover from an unsafe policy, both with an arbitrary user-specified probability, and to establish the asymptotic convergence to the set of KKT points of the RL problem almost surely. Simulations on a navigation example and the cart-pole system illustrate the superior performance of RSGF-RL with respect to the state of the art.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 45 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

22 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Anytime Safe Reinforcement Learning23 Apr
  2. 2024Sampling-based Safe Reinforcement Learning for Nonlinear Dynamical Systems6 Mar

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Convex optimizationcited by 2,515
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  5. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  6. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  7. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  8. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  9. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  11. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  12. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
8 more
  1. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  2. 2023Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processescited by 26
  3. 2021Doubly Robust Off-Policy Actor-Critic: Convergence and Optimalitycited by 21
  4. 2021Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processescited by 17
  5. 2022Control Barrier Function Based Design of Gradient Flows for Constrained Nonlinear Programmingcited by 8
  6. 2023Anytime-Constrained Reinforcement Learningcited by 6
  7. 2021On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correctioncited by 5
  8. 2023Regularity Properties of Optimization-Based Controllerscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints
  2. 2025Online Optimization for Offline Safe Reinforcement Learning
  3. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  5. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
  6. 2026Robust Regularized Policy Iteration under Transition Uncertainty
  7. 2024Off-Policy Primal-Dual Safe Reinforcement Learning
  8. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality