mlsanityData to 7 Oct 2026

Off-Policy Primal-Dual Safe Reinforcement Learning

arXiv 2401.14758 ↗First posted 26 Jan 2024cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 26 Jan 2024.

Builds on

27 of 43 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2018Reward Constrained Policy Optimizationcited by 397
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  11. 2019Batch Policy Learning under Constraintscited by 242
  12. 2021Reinforcement learning based recommender systems: A surveycited by 187
15 more
  1. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  2. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 144
  3. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 128
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  5. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  6. 2019IPO: Interior-point Policy Optimization under Constraintscited by 119
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  8. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  10. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 69
  11. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  12. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  13. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 55
  14. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  15. 2023Safe Offline Reinforcement Learning with Real-Time Budget Constraintscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 8.

13 papers.

Sort:

By area:

  1. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  2. 2025Return Capping: Sample-Efficient CVaR Policy Gradient Optimisationcited by 5
  3. 2026How Does the Lagrangian Guide Safe Reinforcement Learning through Diffusion Models?cited by 4
  4. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  5. 2026Off-Policy Safe Reinforcement Learning with Constrained Optimistic Explorationcited by 2
  6. 2025Don't Trade Off Safety: Diffusion Regularization for Constrained Offline RLcited by 0
  7. 2025Situational-Constrained Sequential Resources Allocation via Reinforcement Learningcited by 0
  8. 2026COP-Q: Safety-First Reinforcement Learning for Robot Control via Cholesky-Ordered Projectioncited by 0
  9. 2026Utility-Constrained Policy Optimizationcited by 0
  10. 2026Boundary-Seeking Policy Gradient for Safe Reinforcement Learningcited by 0
  11. 2026Constrained Flow Policy Updates: A Generalized Schr"odinger Bridge Viewcited by 0
  12. 2026Safe Score Matching: Diffusion Policies with Hamilton-Jacobi Reachability for Online Safe Reinforcement Learningcited by 0
1 more
  1. 2026VERA: Verifiable Feasibility Representations with Counterfactual Credit for Constrained Multi-Agent Controlcited by 0