mlsanityData to 14 Oct 2026

Adversarial Constrained Policy Optimization: Improving Constrained Reinforcement Learning by Adapting Budgets

arXiv 2410.20786 ↗First posted 28 Oct 2024cs.LG cs.ROcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Constrained reinforcement learning has achieved promising progress in safety-critical fields where both rewards and constraints are considered. However, constrained reinforcement learning methods face challenges in striking the right balance between task performance and constraint satisfaction and it is prone for them to get stuck in over-conservative or constraint violating local minima. In this paper, we propose Adversarial Constrained Policy Optimization (ACPO), which enables simultaneous optimization of reward and the adaptation of cost budgets during training. Our approach divides original constrained problem into two adversarial stages that are solved alternately, and the policy update performance of our algorithm can be theoretically guaranteed. We validate our method through experiments conducted on Safety Gymnasium and quadruped locomotion tasks. Results demonstrate that our algorithm achieves better performances compared to commonly used baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 38 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

29 of 38 references.

References · 29

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  7. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  10. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  11. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  12. 2018Safe Exploration in Continuous Action Spacescited by 247
17 more
  1. 2020Projection-Based Constrained Policy Optimizationcited by 203
  2. 2021Reinforcement learning based recommender systems: A surveycited by 188
  3. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  4. 2023Robot Parkour Learningcited by 149
  5. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  6. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  7. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  8. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  9. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  10. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  11. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  12. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  13. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 47
  14. 2021A Primal-Dual Approach to Constrained Markov Decision Processescited by 42
  15. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  16. 2020Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotioncited by 25
  17. 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 20

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Taming the Adversary: A Cost-to-Disturbance Ratio Approach to Adversarial Reinforcement Learning
  2. 2024Constrained Reinforcement Learning Under Model Mismatch
  3. 2024Absolute State-wise Constrained Policy Optimization: High-Probability State-wise Constraints Satisfaction
  4. 2025Adversarial Reinforcement Learning: A Duality-Based Approach To Solving Optimal Control Problems
  5. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation
  6. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning
  7. 2025Incentivizing Safer Actions in Policy Optimization for Constrained Reinforcement Learning
  8. 2025Constraint-Aware Reinforcement Learning via Adaptive Action Scaling