mlsanityData to 14 Oct 2026

Conservative Safety Critics for Exploration

arXiv 2010.14497 ↗First posted 27 Oct 2020cs.LG cs.AI cs.ROcited by 113

Not on the map. Why?

Cited by 31 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Safe exploration presents a major challenge in reinforcement learning (RL): when active data collection requires deploying partially trained policies, we must ensure that these policies avoid catastrophically unsafe regions, while still enabling trial and error learning. In this paper, we target the problem of safe exploration in RL by learning a conservative safety estimate of environment states through a critic, and provably upper bound the likelihood of catastrophic failures at every training iteration. We theoretically characterize the tradeoff between safety and policy improvement, show that the safety constraints are likely to be satisfied with high probability during training, derive provable convergence guarantees for our approach, which is no worse asymptotically than standard RL, and demonstrate the efficacy of the proposed approach on a suite of challenging navigation, manipulation, and locomotion tasks. Empirically, we show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates during training than prior methods. Videos are at this url https://sites.google.com/view/conservative-safety-critics/home

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 24; 2023: 29; 2024: 19; 2025: 12; 2026: 10.

113 papers.

Sort:

By area:

  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  2. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 371
  3. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 181
  4. 2023The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systemscited by 88
  5. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  6. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  7. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  8. 2022Reachability Constrained Reinforcement Learningcited by 59
  9. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  10. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  11. 2022Constrained Policy Optimization via Bayesian World Modelscited by 57
  12. 2023State-wise Safe Reinforcement Learning: A Surveycited by 55
38 more
  1. 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
  2. 2021Safe Reinforcement Learning Using Advantage-Based Interventioncited by 47
  3. 2023Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotioncited by 47
  4. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  5. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 39
  6. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  7. 2022Safe Reinforcement Learning for Legged Locomotioncited by 35
  8. 2021Safe Driving via Expert Guided Policy Optimizationcited by 31
  9. 2021Direct Behavior Specification via Constrained Reinforcement Learningcited by 31
  10. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  11. 2022Sustainable Online Reinforcement Learning for Auto-biddingcited by 29
  12. 2022ISAACS: Iterative Soft Adversarial Actor-Critic for Safetycited by 28
  13. 2022Safe Reinforcement Learning Using Black-Box Reachability Analysiscited by 26
  14. 2021GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Modelcited by 25
  15. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 23
  16. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 21
  17. 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 21
  18. 2022CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learningcited by 20
  19. 2021Value Iteration in Continuous Actions, States and Timecited by 18
  20. 2022Safe Model-Based Reinforcement Learning with an Uncertainty-Aware Reachability Certificatecited by 18
  21. 2023Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learningcited by 18
  22. 2023AutoCost: Evolving Intrinsic Cost for Zero-violation Reinforcement Learningcited by 17
  23. 2024Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imaginationcited by 16
  24. 2021Safe Exploration by Solving Early Terminated MDPcited by 15
  25. 2023A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviorscited by 15
  26. 2024SRL-VIC: A Variable Stiffness-Based Safe Reinforcement Learning for Contact-Rich Robotic Taskscited by 15
  27. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  28. 2021Safe Autonomous Racing via Approximate Reachability on Ego-visioncited by 13
  29. 2023Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functionscited by 13
  30. 2022A Single-Loop Gradient Descent and Perturbed Ascent Algorithm for Nonconvex Functional Constrained Optimizationcited by 12
  31. 2023Risk-Averse Model Uncertainty for Distributionally Robust Safe Reinforcement Learningcited by 12
  32. 2023Probabilistic Constraint for Safety-Critical Reinforcement Learningcited by 11
  33. 2023Efficient Off-Policy Safe Reinforcement Learning Using Trust Region Conditional Value at Riskcited by 11
  34. 2022SafeAPT: Safe Simulation-to-Real Robot Learning using Diverse Policies Learned in Simulationcited by 10
  35. 2023Gradient Shaping for Multi-Constraint Safe Reinforcement Learningcited by 10
  36. 2024RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashescited by 10
  37. 2021Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shieldingcited by 9
  38. 2023Optimal Transport Perturbations for Safe Reinforcement Learning with Robustness Guaranteescited by 9

50 most cited of 113 All 113 (JSON)