mlsanityData to 14 Oct 2026

Learning to be Safe: Deep RL with a Safety Critic

arXiv 2010.14603 ↗First posted 27 Oct 2020cs.LG cs.ROcited by 116

Not on the map. Why?

Cited by 39 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Safety is an essential component for deploying reinforcement learning (RL) algorithms in real-world scenarios, and is critical during the learning process itself. A natural first approach toward safe RL is to manually specify constraints on the policy's behavior. However, just as learning has enabled progress in large-scale development of AI systems, learning safety specifications may also be necessary to ensure safety in messy open-world environments where manual safety specifications cannot scale. Akin to how humans learn incrementally starting in child-safe environments, we propose to learn how to be safe in one set of tasks and environments, and then use that learned intuition to constrain future behaviors when learning new, modified tasks. We empirically study this form of safety-constrained transfer learning in three challenging domains: simulated navigation, quadruped locomotion, and dexterous in-hand manipulation. In comparison to standard deep RL techniques and prior approaches to safe RL, we find that our method enables the learning of new tasks and in new environments with both substantially fewer safety incidents, such as falling or dropping an object, and faster, more stable learning. This suggests a path forward not only for safer RL systems, but also for more effective RL systems.

Key facts. Cited by 116 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 18; 2023: 19; 2024: 29; 2025: 22; 2026: 9.

116 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  3. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 213
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  6. 2023The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systemscited by 88
  7. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  8. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  9. 2025Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysiscited by 45
  10. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 39
  11. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  12. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 38
38 more
  1. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  2. 2022Safe Reinforcement Learning for Legged Locomotioncited by 35
  3. 2021Safe Driving via Expert Guided Policy Optimizationcited by 31
  4. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  5. 2022Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guaranteescited by 27
  6. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 23
  7. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 23
  8. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 22
  9. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  10. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 21
  11. 2022Back to the Manifold: Recovering from Out-of-Distribution Statescited by 20
  12. 2022GoSafeOpt: Scalable Safe Exploration for Global Optimization of Dynamical Systemscited by 17
  13. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 17
  14. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  15. 2021Safe Exploration by Solving Early Terminated MDPcited by 15
  16. 2024Constrained Reinforcement Learning with Smoothed Log Barrier Functioncited by 15
  17. 2024SRL-VIC: A Variable Stiffness-Based Safe Reinforcement Learning for Contact-Rich Robotic Taskscited by 15
  18. 2021Safe Autonomous Racing via Approximate Reachability on Ego-visioncited by 13
  19. 2023Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functionscited by 13
  20. 2024Verification-Guided Shielding for Deep Reinforcement Learningcited by 13
  21. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  22. 2024Safety Filters for Black-Box Dynamical Systems by Learning Discriminating Hyperplanescited by 12
  23. 2025SafeFlowMatcher: Safe and Fast Planning using Flow Matching with Control Barrier Functionscited by 12
  24. 2022Joint Differentiable Optimization and Verification for Certified Reinforcement Learningcited by 11
  25. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 11
  26. 2024RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashescited by 10
  27. 2021Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shieldingcited by 9
  28. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9
  29. 2023Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertaintycited by 9
  30. 2024Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policycited by 9
  31. 2021MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerancecited by 8
  32. 2024Learning Control Barrier Functions and their application in Reinforcement Learning: A Surveycited by 8
  33. 2025Hybrid Action Based Reinforcement Learning for Multi-Objective Compatible Autonomous Drivingcited by 8
  34. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  35. 2021Improving Safety in Deep Reinforcement Learning using Unsupervised Action Planningcited by 7
  36. 2022Provable Safe Reinforcement Learning with Binary Feedbackcited by 7
  37. 2023Reinforcement Learning by Guided Safe Explorationcited by 7
  38. 2024Analyzing Adversarial Inputs in Deep Reinforcement Learningcited by 7

50 most cited of 116 All 116 (JSON)