mlsanityData to 14 Oct 2026

Penalized Proximal Policy Optimization for Safe Reinforcement Learning

arXiv 2205.11814 ↗First posted 24 May 2022cs.LG cs.AI math.OCcited by 68

Not on the map. Why?

Cited by 44 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Safe reinforcement learning aims to learn the optimal policy while satisfying safety constraints, which is essential in real-world applications. However, current algorithms still struggle for efficient policy updates with hard constraint satisfaction. In this paper, we propose Penalized Proximal Policy Optimization (P3O), which solves the cumbersome constrained policy iteration via a single minimization of an equivalent unconstrained problem. Specifically, P3O utilizes a simple-yet-effective penalty function to eliminate cost constraints and removes the trust-region constraint by the clipped surrogate objective. We theoretically prove the exactness of the proposed method with a finite penalty factor and provide a worst-case analysis for approximate error when evaluated on sample trajectories. Moreover, we extend P3O to more challenging multi-constraint and multi-agent scenarios which are less studied in previous work. Extensive experiments show that P3O outperforms state-of-the-art algorithms with respect to both reward improvement and constraint satisfaction on a set of constrained locomotive tasks.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 24 May 2022.

Builds on

15 of 32 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  10. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  11. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  12. 2020Projection-Based Constrained Policy Optimizationcited by 203
3 more
  1. 2019Constrained Reinforcement Learning Has Zero Duality Gapcited by 151
  2. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  3. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 9; 2024: 21; 2025: 17; 2026: 15.

68 papers.

Sort:

By area:

  1. 2023Learning to Search Feasible and Infeasible Regions of Routing Problems with Flexible Neural k-Optcited by 81
  2. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  3. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  4. 2025RSL-RL: A Learning Library for Robotics Researchcited by 49
  5. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 47
  6. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  7. 2023Iterative Reachability Estimation for Safe Reinforcement Learningcited by 37
  8. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  9. 2024CaT: Constraints as Terminations for Legged Locomotion Reinforcement Learningcited by 23
  10. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 22
  11. 2024Stage-Wise Reward Shaping for Acrobatic Robots: A Constrained Multi-Objective Reinforcement Learning Approachcited by 13
  12. 2023Probabilistic Constraint for Safety-Critical Reinforcement Learningcited by 11
38 more
  1. 2024A Review of Safe Reinforcement Learning Methods for Modern Power Systemscited by 11
  2. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 11
  3. 2024Embedding Safety into RL: A New Take on Trust Region Methodscited by 10
  4. 2024Learning Whole-Body Loco-Manipulation for Omni-Directional Task Space Pose Tracking with a Wheeled-Quadrupedal-Manipulatorcited by 10
  5. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  6. 2023Dynamic Routing for Integrated Satellite-Terrestrial Networks: A Constrained Multi-Agent Reinforcement Learning Approachcited by 9
  7. 2022SafeRL-Kit: Evaluating Efficient Reinforcement Learning Methods for Safe Autonomous Drivingcited by 7
  8. 2023Reduced Policy Optimization for Continuous Control with Hard Constraintscited by 6
  9. 2025HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agentscited by 6
  10. 2022Policy Gradients for Probabilistic Constrained Reinforcement Learningcited by 5
  11. 2024Spectral-Risk Safe Reinforcement Learning with Convergence Guaranteescited by 5
  12. 2024Exterior Penalty Policy Optimization with Penalty Metric Network under Constraintscited by 5
  13. 2024Action Mapping for Reinforcement Learning in Continuous Environments with Constraintscited by 5
  14. 2024Adaptive Primal-Dual Method for Safe Reinforcement Learningcited by 4
  15. 2024Long and Short-Term Constraints Driven Safe Reinforcement Learning for Autonomous Drivingcited by 4
  16. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  17. 2022Safe Reinforcement Learning with Contrastive Risk Predictioncited by 3
  18. 2024Safe Reinforcement Learning for Power System Control: A Reviewcited by 3
  19. 2024Flipping-based Policy for Chance-Constrained Markov Decision Processescited by 3
  20. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  21. 2024FAWAC: Feasibility Informed Advantage Weighted Regression for Persistent Safety in Offline Reinforcement Learningcited by 3
  22. 2025Designing Control Barrier Function via Probabilistic Enumeration for Safe Reinforcement Learning Navigationcited by 3
  23. 2025Central Path Proximal Policy Optimizationcited by 3
  24. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  25. 2023A Multiplicative Value Function for Safe and Efficient Reinforcement Learningcited by 2
  26. 2024e-COP : Episodic Constrained Optimization of Policiescited by 2
  27. 2024Vision-driven UAV River Following: Benchmarking with Safe Reinforcement Learningcited by 2
  28. 2025Fairness Aware Reinforcement Learning via Proximal Policy Optimizationcited by 2
  29. 2025End-to-End Humanoid Robot Safe and Comfortable Locomotion Policycited by 2
  30. 2026CRAX: Fast Safe Reinforcement Learning Benchmarkingcited by 2
  31. 2024Simplex Decomposition for Portfolio Allocation Constraints in Reinforcement Learningcited by 1
  32. 2024Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learningcited by 1
  33. 2025DAPPER: Discriminability-Aware Policy-to-Policy Preference-Based Reinforcement Learning for Query-Efficient Robot Skill Acquisitioncited by 1
  34. 2025Exchange Policy Optimization Algorithm for Semi-Infinite Safe Reinforcement Learningcited by 1
  35. 2026Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Explorationcited by 1
  36. 2026ECO: Energy-Constrained Optimization with Reinforcement Learning for Humanoid Walkingcited by 1
  37. 2026Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertaintycited by 1
  38. 2026Augmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learningcited by 1

50 most cited of 68 All 68 (JSON)