mlsanityData to 7 Oct 2026

CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee

arXiv 2011.05869 ↗First posted 11 Nov 2020cs.LG stat.MLcited by 128

Not on the map. Why?

Abstract (arXiv)

In safe reinforcement learning (SRL) problems, an agent explores the environment to maximize an expected total reward and meanwhile avoids violation of certain constraints on a number of expected total costs. In general, such SRL problems have nonconvex objective functions subject to multiple nonconvex constraints, and hence are very challenging to solve, particularly to provide a globally optimal policy. Many popular SRL algorithms adopt a primal-dual structure which utilizes the updating of dual variables for satisfying the constraints. In contrast, we propose a primal approach, called constraint-rectified policy optimization (CRPO), which updates the policy alternatingly between objective improvement and constraint satisfaction. CRPO provides a primal-type algorithmic framework to solve SRL problems, where each policy update can take any variant of policy optimization step. To demonstrate the theoretical performance of CRPO, we adopt natural policy gradient (NPG) for each policy update step and show that CRPO achieves an O(1/√T) convergence rate to the global optimal policy in the constrained policy set and an O(1/√T) error bound on constraint satisfaction. This is the first finite-time analysis of primal SRL algorithms with global optimality guarantee. Our empirical results demonstrate that CRPO can outperform the existing primal-dual baseline algorithms significantly.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 13; 2023: 24; 2024: 30; 2025: 22; 2026: 30.

128 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  3. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  4. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 54
  5. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 46
  6. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  7. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  8. 2022Near-Optimal Sample Complexity Bounds for Constrained MDPscited by 36
  9. 2021A Dual Approach to Constrained Markov Decision Processes with Entropy Regularizationcited by 33
  10. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 33
  11. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  12. 2021Multi-Agent Constrained Policy Optimisationcited by 32
38 more
  1. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 29
  2. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
  3. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
  4. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  5. 2023Long-Term Fairness with Unknown Dynamicscited by 25
  6. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilizationcited by 25
  7. 2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 22
  8. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  9. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
  10. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
  11. 2022Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithmcited by 19
  12. 2021Concave Utility Reinforcement Learning with Zero-Constraint Violationscited by 18
  13. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15
  14. 2023Efficient Gradient Approximation Method for Constrained Bilevel Optimizationcited by 15
  15. 2023A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraintscited by 15
  16. 2021Finite-Time Complexity of Online Primal-Dual Natural Actor-Critic Algorithm for Constrained Markov Decision Processescited by 14
  17. 2022Robust Constrained Reinforcement Learningcited by 14
  18. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  19. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  20. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  21. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
  22. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
  23. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  24. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 11
  25. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 11
  26. 2023A safe exploration approach to constrained Markov decision processescited by 10
  27. 2024Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learningcited by 10
  28. 2025Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspectivecited by 10
  29. 2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9
  30. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  31. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  32. 2024Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithmcited by 9
  33. 2024Beyond the Edge: An Advanced Exploration of Reinforcement Learning for Mobile Edge Computing, its Applications, and Future Research Trajectoriescited by 9
  34. 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learningcited by 9
  35. 2022Towards Painless Policy Optimization for Constrained MDPscited by 8
  36. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 8
  37. 2025Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithmcited by 8
  38. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8

50 most cited of 128 All 128 (JSON)