# Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1901.02219 · HTML: https://mlsanity.com/paper/1901.02219/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2019.

## Built on by (most cited first, 15 of 15)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Full-Spectrum Out-of-Distribution Detection](https://mlsanity.com/paper/2204.05306/) (2022)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learning](https://mlsanity.com/paper/2112.02694/) (2021)
- [Semi-supervised Deep Learning for Image Classification with Distribution Mismatch: A Survey](https://mlsanity.com/paper/2203.00190/) (2022)
- [Statistical Context Detection for Deep Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.19047/) (2024)
- [Learning to Forecast Aleatoric and Epistemic Uncertainties over Long Horizon Trajectories](https://mlsanity.com/paper/2302.08669/) (2023)
- [Mutual Information Tracks Policy Coherence in Reinforcement Learning](https://mlsanity.com/paper/2509.10423/) (2025)
- [How to Enable Uncertainty Estimation in Proximal Policy Optimization](https://mlsanity.com/paper/2210.03649/) (2022)
- [Dealing with Distribution Mismatch in Semi-supervised Deep Learning for Covid-19 Detection Using Chest X-ray Images: A Novel Approach Using Feature Densities](https://mlsanity.com/paper/2109.00889/) (2021)
- [Disentangling Uncertainty for Safe Social Navigation using Deep Reinforcement Learning](https://mlsanity.com/paper/2409.10655/) (2024)
- [When to Trust Your Data: Enhancing Dyna-Style Model-Based Reinforcement Learning With Data Filter](https://mlsanity.com/paper/2410.12160/) (2024)
- [An out-of-distribution discriminator based on Bayesian neural network epistemic uncertainty](https://mlsanity.com/paper/2210.10780/) (2022)
- [DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Critic](https://mlsanity.com/paper/2310.17173/) (2023)
- [Criteria for Uncertainty-based Corner Cases Detection in Instance Segmentation](https://mlsanity.com/paper/2404.11266/) (2024)

## Abstract (arXiv)

We consider the problem of detecting out-of-distribution (OOD) samples in deep reinforcement learning. In a value based reinforcement learning setting, we propose to use uncertainty estimation techniques directly on the agent's value estimating neural network to detect OOD samples. The focus of our work lies in analyzing the suitability of approximate Bayesian inference methods and related ensembling techniques that generate uncertainty estimates. Although prior work has shown that dropout-based variational inference techniques and bootstrap-based approaches can be used to model epistemic uncertainty, the suitability for detecting OOD samples in deep reinforcement learning remains an open question. Our results show that uncertainty estimation can be used to differentiate in- from out-of-distribution samples. Over the complete training process of the reinforcement learning agents, bootstrap-based approaches tend to produce more reliable epistemic uncertainty estimates, when compared to dropout-based approaches.