Data to 7 Oct 2026

Projection-Based Constrained Policy Optimization

arXiv 2010.03152 ↗First posted 7 Oct 2020cs.LG cs.AI cs.ROcited by 195

Not on the map. Why?

Abstract (arXiv)

We consider the problem of learning control policies that optimize a reward function while satisfying constraints due to considerations of safety, fairness, or other costs. We propose a new algorithm, Projection-Based Constrained Policy Optimization (PCPO). This is an iterative method for optimizing policies in a two-step process: the first step performs a local reward improvement update, while the second step reconciles any constraint violation by projecting the policy back onto the constraint set. We theoretically analyze PCPO and provide a lower bound on reward improvement, and an upper bound on constraint violation, for each policy update. We further characterize the convergence of PCPO based on two different metrics: normltwo norm and Kullback-Leibler divergence. Our empirical results over several control tasks demonstrate that PCPO achieves superior performance, averaging more than 3.5 times less constraint violation and around 15% higher reward compared to state-of-the-art methods.

Key facts. Cited by 195 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 30; 2023: 45; 2024: 40; 2025: 29; 2026: 26.

195 papers.

Sort:

By area:

2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 78
2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
2022Reachability Constrained Reinforcement Learningcited by 52

38 more

2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 47
2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
2021DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learningcited by 34
2021Context-Aware Safe Reinforcement Learning for Non-Stationary Environmentscited by 33
2021Safe Reinforcement Learning with Linear Function Approximationcited by 33
2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 33
2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
2021Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimizationcited by 31
2022Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarkingcited by 31
2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 30
2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 29
2022Safe Reinforcement Learning for Legged Locomotioncited by 29
2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
2022Towards Safe Reinforcement Learning with a Safety Editor Policycited by 28
2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 26
2021Safe Pontryagin Differentiable Programmingcited by 25
2021Density Constrained Reinforcement Learningcited by 25
2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
2022Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopescited by 23
2021Policy Optimization for Constrained MDPs with Provable Fast Global Convergencecited by 22
2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
2021Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimizationcited by 18
2021Faster Algorithm and Sharper Analysis for Constrained Markov Decision Processcited by 18
2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17
2022SABLAS: Learning Safe Control for Black-box Dynamical Systemscited by 17
2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 17
2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 16
2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 16
2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learningcited by 16

50 most cited of 195 All 195 (JSON)