mlsanityData to 14 Oct 2026

Embedding Safety into RL: A New Take on Trust Region Methods

arXiv 2411.02957 ↗First posted 5 Nov 2024cs.LG cs.SY eess.SYcited by 10

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Reinforcement Learning (RL) agents can solve diverse tasks but often exhibit unsafe behavior. Constrained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrifice reward maximization or allow unsafe training. We introduce Constrained Trust Region Policy Optimization (C-TRPO), which reshapes the policy space geometry to ensure trust regions contain only safe policies, guaranteeing constraint satisfaction throughout training. We analyze its theoretical properties and connections to TRPO, Natural Policy Gradient (NPG), and Constrained Policy Optimization (CPO). Experiments show that C-TRPO reduces constraint violations while maintaining competitive returns.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 51 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

32 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning12 Oct
  2. 2024Constrained Reinforcement Learning with Smoothed Log Barrier Function21 Mar

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  7. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  8. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  9. 2016Control Barrier Function Based Quadratic Programs for Safety Critical Systemscited by 287
  10. 2019A Theory of Regularized Markov Decision Processescited by 286
  11. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  12. 2020Projection-Based Constrained Policy Optimizationcited by 203
18 more
  1. 2017A unified view of entropy-regularized Markov decision processescited by 182
  2. 2020Exploration-Exploitation in Constrained MDPscited by 149
  3. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  4. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  5. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  6. 2020Mirror Descent Policy Optimizationcited by 106
  7. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  8. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 68
  9. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  10. 2018Hessian Riemannian gradient flows in convex programmingcited by 26
  11. 2021On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributionscited by 23
  12. 2024Truly No-Regret Learning in Constrained MDPscited by 23
  13. 2022Geometry and convergence of natural policy gradient methodscited by 12
  14. 2024On PI Controllers for Updating Lagrange Multipliers in Constrained Optimizationcited by 12
  15. 2022Invariance Properties of the Natural Gradient in Overparametrised Systemscited by 11
  16. 2022Log Barriers for Safe Black-box Optimization with Application to Safe Reinforcement Learningcited by 11
  17. 2023A safe exploration approach to constrained Markov decision processescited by 10
  18. 2024Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradientscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 21
  2. 2026Safe Exploration via Policy Priorscited by 6
  3. 2025Central Path Proximal Policy Optimizationcited by 3
  4. 2025SPoRt -- Safe Policy Ratio: Certified Training and Deployment of Task Policies in Model-Free RLcited by 1
  5. 2025The Geometry of Nonlinear Reinforcement Learningcited by 1
  6. 2026Constrained Meta Reinforcement Learning with Provable Test-Time Safetycited by 1
  7. 2025SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraintscited by 0
  8. 2026Beyond ESG Scores: Learning Dynamic Constraints for Sequential Portfolio Optimizationcited by 0
  9. 2026CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learningcited by 0
  10. 2026Reachability-Aware Diffusion Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning
  2. 2025A Survey of Safe Reinforcement Learning and Constrained MDPs: A Technical Survey on Single-Agent and Multi-Agent Safety
  3. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  4. 2023Supported Trust Region Optimization for Offline Reinforcement Learning
  5. 2026Utility-Constrained Policy Optimization
  6. 2024Concurrent Learning of Policy and Unknown Safety Constraints in Reinforcement Learning
  7. 2026Evaluation Metrics for Safe Reinforcement Learning
  8. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards