mlsanityData to 7 Oct 2026

CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning

arXiv 2202.07565 ↗First posted 15 Feb 2022cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

Safe reinforcement learning (RL) is still very challenging since it requires the agent to consider both return maximization and safe exploration. In this paper, we propose CUP, a Conservative Update Policy algorithm with a theoretical safety guarantee. We derive the CUP based on the new proposed performance bounds and surrogate functions. Although using bounds as surrogate functions to design safe RL algorithms have appeared in some existing works, we develop them at least three aspects: (i) We provide a rigorous theoretical analysis to extend the surrogate functions to generalized advantage estimator (GAE). GAE significantly reduces variance empirically while maintaining a tolerable level of bias, which is an efficient step for us to design CUP; (ii) The proposed bounds are tighter than existing works, i.e., using the proposed bounds as surrogate functions are better local approximations to the objective and safety constraints. (iii) The proposed CUP provides a non-convex implementation via first-order optimizers, which does not depend on any convex approximation. Finally, extensive experiments show the effectiveness of CUP where the agent satisfies safe constraints. We have opened the source code of CUP at https://github.com/RL-boxes/Safe-RL.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 28 of its 53 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

28 of 53 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2016OpenAI Gymcited by 1,684
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018Reward Constrained Policy Optimizationcited by 397
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
  10. 2019Batch Policy Learning under Constraintscited by 242
  11. 2020Projection-Based Constrained Policy Optimizationcited by 195
  12. 2021Reinforcement learning based recommender systems: A surveycited by 187
16 more
  1. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  2. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  3. 2020Conservative Safety Critics for Explorationcited by 107
  4. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  5. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  6. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 53
  7. 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
  8. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  9. 2020Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertaintycited by 23
  10. 2021Safe Continuous Control with Constrained Model-Based Policy Optimizationcited by 17
  11. 2020Sample Complexity of Policy Gradient Finding Second-Order Stationary Pointscited by 16
  12. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11
  13. 2020Accelerating Safe Reinforcement Learning with Constraint-mismatched Policiescited by 11
  14. 2020Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guaranteecited by 7
  15. 2018A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learningcited by 4
  16. 2020On Convergence of Gradient Expected Sarsa(λ)cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 5; 2025: 3; 2026: 2.

16 papers.

Sort:

By area:

  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  3. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 33
  4. 2023Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPscited by 27
  5. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26
  6. 2024Hamilton-Jacobi Reachability in Reinforcement Learning: A Surveycited by 11
  7. 2025Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspectivecited by 10
  8. 2024Optimal Transport-Assisted Risk-Sensitive Q-Learningcited by 3
  9. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  10. 2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learningcited by 1
  11. 2023Efficient Exploration Using Extra Safety Budget in Constrained Policy Optimizationcited by 0
  12. 2024Autoregressive Policy Optimization for Constrained Allocation Taskscited by 0
4 more
  1. 2024Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgetscited by 0
  2. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewardscited by 0
  3. 2026Fuz-RL: A Fuzzy-Guided Robust Framework for Safe Reinforcement Learning under Uncertaintycited by 0
  4. 2026Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observabilitycited by 0