mlsanityData to 14 Oct 2026

Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation

arXiv 2510.18478 ↗First posted 21 Oct 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Safe and constrained RL

Abstract (arXiv)

Ensuring the safe exploration of reinforcement learning (RL) agents is critical for deployment in real-world systems. Yet existing approaches struggle to strike the right balance: methods that tightly enforce safety often cripple task performance, while those that prioritize reward leave safety constraints frequently violated, producing diffuse cost landscapes that flatten gradients and stall policy improvement. We introduce the Uncertain Safety Critic (USC), a novel approach that integrates uncertainty-aware modulation and refinement into critic training. By concentrating conservatism in uncertain and costly regions while preserving sharp gradients in safe areas, USC enables policies to achieve effective reward-safety trade-offs. Extensive experiments show that USC reduces safety violations by approximately 40% while maintaining competitive or higher rewards, and reduces the error between predicted and true cost gradients by approximately 83%, breaking the prevailing trade-off between safety and performance and paving the way for scalable safe RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 44 references that are in the map. First posted on arXiv 21 Oct 2025.

Builds on

28 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Safe Reinforcement Learning in Black-Box Environments via Adaptive Shielding28 May
  2. 2024A Survey of Constraint Formulations in Safe Reinforcement Learning3 Feb
  3. 2024Learning safety critics via a non-contractive binary bellman operator23 Jan

Other references · 25

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2017Constrained Policy Optimizationcited by 983
  6. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  7. 2017Safe Reinforcement Learning via Shieldingcited by 449
  8. 2018Reward Constrained Policy Optimizationcited by 417
  9. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  10. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  11. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  12. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
13 more
  1. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 143
  2. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  3. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  4. 2020Conservative Safety Critics for Explorationcited by 113
  5. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 103
  6. 2019Safe Policies for Reinforcement Learning via Primal-Dual Methodscited by 79
  7. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 74
  8. 2019Convergent Policy Optimization for Safe Reinforcement Learningcited by 57
  9. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  10. 2022COOL-MC: A Comprehensive Tool for Reinforcement Learning and Model Checkingcited by 19
  11. 2023Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstrationcited by 14
  12. 2021Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Interventioncited by 5
  13. 2022Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistakecited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards
  2. 2025Safety Representations for Safer Policy Learning
  3. 2025Provably Optimal Reinforcement Learning under Safety Filtering
  4. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
  5. 2026Sampling-Based Safe Reinforcement Learning
  6. 2026Safe Exploration via Policy Priors
  7. 2025Constraint-Aware Reinforcement Learning via Adaptive Action Scaling
  8. 2024Safe and Robust Reinforcement Learning: Principles and Practice