mlsanityData to 7 Oct 2026

Constrained Update Projection Approach to Safe Policy Optimization

arXiv 2209.07089 ↗First posted 15 Sep 2022cs.LG cs.AIcited by 55

Not on the map. Why?

Abstract (arXiv)

Safe reinforcement learning (RL) studies problems where an intelligent agent has to not only maximize reward but also avoid exploring unsafe areas. In this study, we propose CUP, a novel policy optimization method based on Constrained Update Projection framework that enjoys rigorous safety guarantee. Central to our CUP development is the newly proposed surrogate functions along with the performance bound. Compared to previous safe RL methods, CUP enjoys the benefits of 1) CUP generalizes the surrogate functions to generalized advantage estimator (GAE), leading to strong empirical performance. 2) CUP unifies performance bounds, providing a better understanding and interpretability for some existing algorithms; 3) CUP provides a non-convex implementation via only first-order optimizers, which does not require any strong approximation on the convexity of the objectives. To validate our CUP method, we compared CUP against a comprehensive list of safe RL baselines on a wide range of tasks. Experiments show the effectiveness of CUP both in terms of reward and safety constraint satisfaction. We have opened the source code of CUP at this link https://github.com/zmsn-2077/ CUP-safe-rl.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 29 of its 64 references that are in the map. First posted on arXiv 15 Sep 2022.

Builds on

29 of 64 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2021Convex optimizationcited by 2,479
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018Reward Constrained Policy Optimizationcited by 397
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  8. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  9. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2018Safe Exploration in Continuous Action Spacescited by 237
  12. 2020Projection-Based Constrained Policy Optimizationcited by 195
17 more
  1. 2021Reinforcement learning based recommender systems: A surveycited by 187
  2. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  3. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  4. 2020Conservative Safety Critics for Explorationcited by 107
  5. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  6. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  7. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 53
  8. 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
  9. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  10. 2020Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertaintycited by 23
  11. 2019Temporal Logic Guided Safe Reinforcement Learning Using Control Barrier Functionscited by 20
  12. 2019Policy Optimization with Stochastic Mirror Descentcited by 17
  13. 2020Sample Complexity of Policy Gradient Finding Second-Order Stationary Pointscited by 16
  14. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11
  15. 2020Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guaranteecited by 7
  16. 2018A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learningcited by 4
  17. 2020On Convergence of Gradient Expected Sarsa(λ)cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 17; 2025: 10; 2026: 19.

55 papers.

Sort:

By area:

  1. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  2. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  3. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  4. 2024Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulationcited by 21
  5. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18
  6. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularizationcited by 14
  7. 2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
  8. 2024Verification-Guided Shielding for Deep Reinforcement Learningcited by 13
  9. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 9
  10. 2024Enhancing Efficiency of Safe Reinforcement Learning via Sample Manipulationcited by 8
  11. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  12. 2023Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learningcited by 7
38 more
  1. 2023Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learningcited by 6
  2. 2023Reduced Policy Optimization for Continuous Control with Hard Constraintscited by 5
  3. 2024Safety through feedback in Constrained RLcited by 5
  4. 2024Exterior Penalty Policy Optimization with Penalty Metric Network under Constraintscited by 5
  5. 2023Feasible Policy Iteration for Safe Reinforcement Learningcited by 4
  6. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 4
  7. 2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
  8. 2026Extreme Value Policy Optimization for Safe Reinforcement Learningcited by 3
  9. 2024FlagVNE: A Flexible and Generalizable Reinforcement Learning Framework for Network Resource Allocationcited by 2
  10. 2024Safe Reinforcement Learning using Finite-Horizon Gradient-based Estimationcited by 2
  11. 2025Adaptive Shielding for Safe Reinforcement Learning under Hidden-Parameter Dynamics Shiftscited by 2
  12. 2025Safe In-Context Reinforcement Learningcited by 2
  13. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  14. 2024Diffusion Models for Offline Multi-agent Reinforcement Learning with Safety Constraintscited by 1
  15. 2024Constrained Reinforcement Learning for Safe Heat Pump Controlcited by 1
  16. 2025Proactive Constrained Policy Optimization with Preemptive Penaltycited by 1
  17. 2026Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Explorationcited by 1
  18. 2026Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbedcited by 1
  19. 2026SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancycited by 1
  20. 2026Vision--Language Signals in Constrained RL: Safety Gains Without Anticipationcited by 1
  21. 2023A General Perspective on Objectives of Reinforcement Learningcited by 0
  22. 2024GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Modelcited by 0
  23. 2024mathrmE2CFD: Towards Effective and Efficient Cost Function Design for Safe Reinforcement Learning via Large Language Modelcited by 0
  24. 2024Autoregressive Policy Optimization for Constrained Allocation Taskscited by 0
  25. 2024Reinfier and Reintrainer: Verification and Interpretation-Driven Safe Deep Reinforcement Learning Frameworkscited by 0
  26. 2024From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learningcited by 0
  27. 2025TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labelscited by 0
  28. 2025A universal policy wrapper with guaranteescited by 0
  29. 2025Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learningcited by 0
  30. 2025KFCPO: Kronecker-Factored Approximated Constrained Policy Optimizationcited by 0
  31. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraintscited by 0
  32. 2026Enhance the Safety in Reinforcement Learning by ADRC Lagrangian Methodscited by 0
  33. 2026A Safety-Constrained Reinforcement Learning Framework for Reliable Wireless Autonomycited by 0
  34. 2026Constraint-Enhanced Reinforcement Learning Based on Dynamic Decoupled Spherical Radial Squashingcited by 0
  35. 2026Kernel-Based Safe Exploration in Deep Reinforcement Learningcited by 0
  36. 2026Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learningcited by 0
  37. 2026COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projectioncited by 0
  38. 2026Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Modelscited by 0

50 most cited of 55 All 55 (JSON)