mlsanityData to 14 Oct 2026

GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model

arXiv 2406.03912 ↗First posted 6 Jun 2024cs.AI cs.LG cs.ROcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safe Reinforcement Learning (SRL) aims to realize a safe learning process for Deep Reinforcement Learning (DRL) algorithms by incorporating safety constraints. However, the efficacy of SRL approaches often relies on accurate function approximations, which are notably challenging to achieve in the early learning stages due to data insufficiency. To address this issue, we introduce in this work a novel Generalizable Safety enhancer (GenSafe) that is able to overcome the challenge of data insufficiency and enhance the performance of SRL approaches. Leveraging model order reduction techniques, we first propose an innovative method to construct a Reduced Order Markov Decision Process (ROMDP) as a low-dimensional approximator of the original safety constraints. Then, by solving the reformulated ROMDP-based constraints, GenSafe refines the actions of the agent to increase the possibility of constraint satisfaction. Essentially, GenSafe acts as an additional safety layer for SRL algorithms. We evaluate GenSafe on multiple SRL approaches and benchmark problems. The results demonstrate its capability to improve safety performance, especially in the early learning phases, while maintaining satisfactory task performance. Our proposed GenSafe not only offers a novel measure to augment existing SRL methods but also shows broad compatibility with various SRL algorithms, making it applicable to a wide range of systems and SRL problems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 88 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

48 of 88 references.

References · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2017Constrained Policy Optimizationcited by 983
  5. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  6. 2018Reward Constrained Policy Optimizationcited by 417
  7. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  8. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  9. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  10. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  11. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  12. 2018Safe Exploration in Continuous Action Spacescited by 247
36 more
  1. 2020Projection-Based Constrained Policy Optimizationcited by 203
  2. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  3. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  4. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  5. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 163
  6. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  7. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  8. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  9. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  10. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  11. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  12. 2020Safe Reinforcement Learning in Constrained Markov Decision Processescited by 115
  13. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 111
  14. 2014Algorithms for CVaR Optimization in MDPscited by 99
  15. 2019Safe Reinforcement Learning Using Robust MPCcited by 85
  16. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  17. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 72
  18. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  19. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 62
  20. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  21. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  22. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  23. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  24. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
  25. 2020Learning Off-Policy with Online Planningcited by 53
  26. 2016Safe Learning of Regions of Attraction for Uncertain, Nonlinear Systems with Gaussian Processescited by 39
  27. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 36
  28. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 30
  29. 2022Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithmcited by 29
  30. 2021Conservative and Adaptive Penalty for Model-Based Safe Reinforcement Learningcited by 16
  31. 2020Learning for MPC with Stability & Safety Guaranteescited by 14
  32. 2023Sample-Efficient and Safe Deep Reinforcement Learning via Reset Deep Ensemble Agentscited by 12
  33. 2020Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Methodcited by 7
  34. 2020Learning a Low-dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systemscited by 6
  35. 2022Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processescited by 3
  36. 2019Optimal Reduced-order Modeling of Bipedal Locomotioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  2. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints
  3. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
  4. 2026Evaluation Metrics for Safe Reinforcement Learning
  5. 2024Safety Optimized Reinforcement Learning via Multi-Objective Policy Optimization
  6. 2024Long-term Safe Reinforcement Learning with Binary Feedback
  7. 2025CBF-RL: Safety Filtering Reinforcement Learning in Training with Control Barrier Functions
  8. 2024Safe Reinforcement Learning for Constrained Markov Decision Processes with Stochastic Stopping Time