# Safe Reinforcement Learning with Model Uncertainty Estimates

arXiv: https://arxiv.org/abs/1810.08700 · HTML: https://mlsanity.com/paper/1810.08700/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2018.

## Built on by (most cited first, 30 of 72)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2206.01558/) (2022)
- [Sim-to-Lab-to-Real: Safe Reinforcement Learning with Shielding and Generalization Guarantees](https://mlsanity.com/paper/2201.08355/) (2022)
- [Density Constrained Reinforcement Learning](https://mlsanity.com/paper/2106.12764/) (2021)
- [Adaptive Bounding Box Uncertainties via Two-Step Conformal Prediction](https://mlsanity.com/paper/2403.07263/) (2024)
- [Representation Uncertainty in Self-Supervised Learning as Variational Inference](https://mlsanity.com/paper/2203.11437/) (2022)
- [Safe Reinforcement Learning of Dynamic High-Dimensional Robotic Tasks: Navigation, Manipulation, Interaction](https://mlsanity.com/paper/2209.13308/) (2022)
- [Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation](https://mlsanity.com/paper/2104.03111/) (2021)
- [Generating Physically-Consistent Satellite Imagery for Climate Visualizations](https://mlsanity.com/paper/2104.04785/) (2021)
- [Safe Chance Constrained Reinforcement Learning for Batch Process Control](https://mlsanity.com/paper/2104.11706/) (2021)
- [Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learning](https://mlsanity.com/paper/2112.02694/) (2021)
- [Safety Enhancement for Deep Reinforcement Learning in Autonomous Separation Assurance](https://mlsanity.com/paper/2105.02331/) (2021)
- [Uncertainty Prioritized Experience Replay](https://mlsanity.com/paper/2506.09270/) (2025)
- [Adaptive Risk-Tendency: Nano Drone Navigation in Cluttered Environments with Distributional Reinforcement Learning](https://mlsanity.com/paper/2203.14749/) (2022)
- [Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation](https://mlsanity.com/paper/2503.05238/) (2025)
- [MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2109.10552/) (2021)
- [Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention](https://mlsanity.com/paper/2111.05819/) (2021)
- [SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Driving](https://mlsanity.com/paper/2211.04009/) (2022)
- [Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning](https://mlsanity.com/paper/2306.06754/) (2023)
- [Monte Carlo Tree Search in the Presence of Transition Uncertainty](https://mlsanity.com/paper/2312.11348/) (2023)
- [Reinforcement Learning with Ensemble Model Predictive Safety Certification](https://mlsanity.com/paper/2402.04182/) (2024)
- [Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Duality](https://mlsanity.com/paper/2508.17448/) (2025)
- [XAI-N: Sensor-based Robot Navigation using Expert Policies and Decision Trees](https://mlsanity.com/paper/2104.10818/) (2021)
- [Lyapunov-based uncertainty-aware safe reinforcement learning](https://mlsanity.com/paper/2107.13944/) (2021)
- [A Verification Framework for Certifying Learning-Based Safety-Critical Aviation Systems](https://mlsanity.com/paper/2205.04590/) (2022)
- [Variational Meta Reinforcement Learning for Social Robotics](https://mlsanity.com/paper/2206.03211/) (2022)

## Abstract (arXiv)

Many current autonomous systems are being designed with a strong reliance on black box predictions from deep neural networks (DNNs). However, DNNs tend to be overconfident in predictions on unseen data and can give unpredictable results for far-from-distribution test data. The importance of predictions that are robust to this distributional shift is evident for safety-critical applications, such as collision avoidance around pedestrians. Measures of model uncertainty can be used to identify unseen data, but the state-of-the-art extraction methods such as Bayesian neural networks are mostly intractable to compute. This paper uses MC-Dropout and Bootstrapping to give computationally tractable and parallelizable uncertainty estimates. The methods are embedded in a Safe Reinforcement Learning framework to form uncertainty-aware navigation around pedestrians. The result is a collision avoidance policy that knows what it does not know and cautiously avoids pedestrians that exhibit unseen behavior. The policy is demonstrated in simulation to be more robust to novel observations and take safer actions than an uncertainty-unaware baseline.