CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
On the map: Deep reinforcement learning methods → Safe and constrained RL
Abstract (arXiv)
Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 27 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
19 of 27 references.
Closest ancestors · 3
Same direction, newest first
- 2024Embedding Safety into RL: A New Take on Trust Region Methods5 Nov
- 2024Exterior Penalty Policy Optimization with Penalty Metric Network under Constraints22 Jul
- 2024Adaptive Primal-Dual Method for Safe Reinforcement Learning1 Feb
Other references · 16
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Trust Region Policy Optimizationcited by 2,580
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2017Constrained Policy Optimizationcited by 983
- 2018Reward Constrained Policy Optimizationcited by 417
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
- 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
- 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
- 2020Projection-Based Constrained Policy Optimizationcited by 203
- 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
- 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
- 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
4 more
- 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
- 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
- 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
- 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 22
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Proactive Constrained Policy Optimization with Preemptive Penalty
- 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
- 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction
- 2021Safe Continuous Control with Constrained Model-Based Policy Optimization
- 2024Safe and Balanced: A Framework for Constrained Multi-Objective Reinforcement Learning
- 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
- 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality
- 2025Constraint-Aware Reinforcement Learning via Adaptive Action Scaling