mlsanityData to 14 Oct 2026

Constrained Policy Optimization

arXiv 1705.10528 ↗First posted 30 May 2017cs.LGcited by 983

Not on the map. Why?

Cited by 349 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For many applications of reinforcement learning it can be more convenient to specify both a reward function and constraints, rather than trying to design behavior through the reward function. For example, systems that physically interact with or around humans should satisfy safety constraints. Recent advances in policy search algorithms (Mnih et al., 2016, Schulman et al., 2015, Lillicrap et al., 2016, Levine et al., 2016) have enabled new capabilities in high-dimensional control, but do not consider the constrained setting. We propose Constrained Policy Optimization (CPO), the first general-purpose policy search algorithm for constrained reinforcement learning with guarantees for near-constraint satisfaction at each iteration. Our method allows us to train neural network policies for high-dimensional control while making guarantees about policy behavior all throughout training. Our guarantees are based on a new theoretical result, which is of independent interest: we prove a bound relating the expected returns of two policies to an average divergence between them. We demonstrate the effectiveness of our approach on simulated robot locomotion tasks where the agent must satisfy constraints motivated by safety.

Key facts. Cited by 983 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 2; 2019: 0; 2020: 2; 2021: 103; 2022: 120; 2023: 142; 2024: 160; 2025: 173; 2026: 281.

983 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  4. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  5. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  6. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  7. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  8. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  9. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  10. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  11. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  12. 2021Offline RL Without Off-Policy Evaluationcited by 151
38 more
  1. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  2. 2023Motion Planning for Autonomous Driving: The State of the Art and Future Perspectivescited by 114
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  4. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  5. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  6. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  7. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  8. 2021Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPscited by 80
  9. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  10. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  11. 2022Constrained Variational Policy Optimization for Safe Reinforcement Learningcited by 71
  12. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  13. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  14. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  15. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 69
  16. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  17. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  18. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  19. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  20. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  21. 2022Reachability Constrained Reinforcement Learningcited by 59
  22. 2023Rewarding Chatbots for Real-World Engagement with Millions of Userscited by 59
  23. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 59
  24. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  25. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  26. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  27. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
  28. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  29. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 54
  30. 2021Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challengescited by 50
  31. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  32. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
  33. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  34. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 47
  35. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  36. 2023Behavior Proximal Policy Optimizationcited by 43
  37. 2021Constrained Learning with Non-Convex Lossescited by 42
  38. 2023Safe Reinforcement Learning via Probabilistic Logic Shieldscited by 42

50 most cited of 983 All 983 (JSON)