mlsanityData to 14 Oct 2026

Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning

arXiv 2409.12045 ↗First posted 18 Sep 2024cs.LG cs.ROcited by 3

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Safety is one of the key issues preventing the deployment of reinforcement learning techniques in real-world robots. While most approaches in the Safe Reinforcement Learning area do not require prior knowledge of constraints and robot kinematics and rely solely on data, it is often difficult to deploy them in complex real-world settings. Instead, model-based approaches that incorporate prior knowledge of the constraints and dynamics into the learning framework have proven capable of deploying the learning algorithm directly on the real robot. Unfortunately, while an approximated model of the robot dynamics is often available, the safety constraints are task-specific and hard to obtain: they may be too complicated to encode analytically, too expensive to compute, or it may be difficult to envision a priori the long-term safety requirements. In this paper, we bridge this gap by extending the safe exploration method, ATACOM, with learnable constraints, with a particular focus on ensuring long-term safety and handling of uncertainty. Our approach is competitive or superior to state-of-the-art methods in final performance while maintaining safer behavior during training.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 50 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

41 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applications13 Apr

Other references · 40

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2019Control Barrier Functions: Theory and Applicationscited by 372
  10. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  12. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
28 more
  1. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 297
  2. 2018Safe Exploration in Continuous Action Spacescited by 247
  3. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 145
  4. 2020Provably Efficient Safe Exploration via Primal-Dual Policy Optimizationcited by 134
  5. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  6. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 111
  7. 2021Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functionscited by 94
  8. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
  9. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 80
  10. 2019Learning for Safety-Critical Control with Control Barrier Functionscited by 69
  11. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  12. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  13. 2020Learning Off-Policy with Online Planningcited by 53
  14. 2019Worst Cases Policy Gradientscited by 52
  15. 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
  16. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  17. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 41
  18. 2020SAMBA: Safe Model-Based & Active Reinforcement Learningcited by 30
  19. 2022Safe Control Under Input Limits with Neural Control Barrier Functionscited by 29
  20. 2021Safe Reinforcement Learning with Chance-constrained Model Predictive Controlcited by 20
  21. 2022Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interactioncited by 17
  22. 2015Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regretcited by 15
  23. 2021Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learningcited by 13
  24. 2021Lyapunov Barrier Policy Optimizationcited by 12
  25. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  26. 2024RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashescited by 10
  27. 2018Calculating CVaR and bPOE for Common Probability Distributions With Application to Portfolio Optimization and Density Estimationcited by 4
  28. 2023Feasible Policy Iteration for Safe Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Controlcited by 0
  2. 2026Directional Constraints for Efficient Exploration in Safe Reinforcement Learningcited by 0
  3. 2026Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shieldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Safe Continuous Control with Constrained Model-Based Policy Optimization
  2. 2024A Dynamic Safety Shield for Safe and Efficient Reinforcement Learning of Navigation Tasks
  3. 2026Sampling-Based Safe Reinforcement Learning
  4. 2026CAPSULE: Control-Theoretic Action Perturbations for Safe Uncertainty-Aware Reinforcement Learning
  5. 2025Towards Safe Robot Foundation Models
  6. 2026Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation
  7. 2022SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulation
  8. 2026Imagine to Ensure Safety in Hierarchical Reinforcement Learning