Data to 14 Oct 2026

Convergent Policy Optimization for Safe Reinforcement Learning

arXiv 1910.12156 ↗First posted 26 Oct 2019cs.LG stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

We study the safe reinforcement learning problem with nonlinear function approximation, where policy optimization is formulated as a constrained optimization problem with both the objective and the constraint being nonconvex functions. For such a problem, we construct a sequence of surrogate convex constrained optimization problems by replacing the nonconvex functions locally with convex quadratic functions obtained from policy gradient estimators. We prove that the solutions to these surrogate problems converge to a stationary point of the original nonconvex problem. Furthermore, to extend our theoretical results, we apply our algorithm to examples of optimal control and multi-agent reinforcement learning with safety constraints.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 13; 2023: 10; 2024: 13; 2025: 7; 2026: 1.

57 papers.

Sort:

By area:

2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
2021Safe Reinforcement Learning with Linear Function Approximationcited by 35
2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 29
2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 28
2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 27
2021Safe Pontryagin Differentiable Programmingcited by 26
2021An Adaptive Stochastic Sequential Quadratic Programming with Differentiable Exact Augmented Lagrangianscited by 25
2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 19
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18

38 more

2021Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processescited by 17
2022Robust Constrained Reinforcement Learningcited by 15
2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 15
2022Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systemscited by 14
2022Near-Optimal Regret Bounds for Multi-batch Reinforcement Learningcited by 13
2022A Single-Loop Gradient Descent and Perturbed Ascent Algorithm for Nonconvex Functional Constrained Optimizationcited by 12
2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
2023Provably Safe Reinforcement Learning with Step-wise Violation Constraintscited by 11
2024A Review of Safe Reinforcement Learning Methods for Modern Power Systemscited by 11
2021Safe Distributional Reinforcement Learningcited by 10
2022Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraintscited by 10
2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 10
2021Almost Optimal Batch-Regret Tradeoff for Batch Linear Contextual Banditscited by 9
2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9
2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 9
2022Towards Painless Policy Optimization for Constrained MDPscited by 8
2024Constrained Reinforcement Learning Under Model Mismatchcited by 8
2023The AI Revolution: Opportunities and Challenges for the Finance Sectorcited by 6
2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
2021Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learningcited by 5
2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Timecited by 5
2021Model-Based Safe Reinforcement Learning with Time-Varying State and Control Constraints: An Application to Intelligent Vehiclescited by 4
2022Sampling Through the Lens of Sequential Decision Makingcited by 4
2021Risk Conditioned Neural Motion Planningcited by 3
2024Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learningcited by 3
2024Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learningcited by 2
2021DeCOM: Decomposed Policy for Constrained Cooperative Multi-Agent Reinforcement Learningcited by 1
2023Learning to Recover for Safe Reinforcement Learningcited by 1
2024Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergencecited by 1
2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learningcited by 1
2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
2021Safe Learning of Uncertain Environmentscited by 0
2023Human Machine Co-adaption Interface via Cooperation Markov Decision Process Systemcited by 0
2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
2023A Single-Loop Deep Actor-Critic Algorithm for Constrained Reinforcement Learning with Provable Convergencecited by 0
2024Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learningcited by 0
2024GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Modelcited by 0

50 most cited of 57 All 57 (JSON)