# Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2112.02694 · HTML: https://mlsanity.com/paper/2112.02694/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches](https://mlsanity.com/paper/1803.04386/) (2018)
- [Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancy](https://mlsanity.com/paper/1908.04951/) (2019)
- [DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks](https://mlsanity.com/paper/1906.04569/) (2019)
- [Safe Reinforcement Learning with Model Uncertainty Estimates](https://mlsanity.com/paper/1810.08700/) (2018)
- [Classification Uncertainty of Deep Neural Networks Based on Gradient Information](https://mlsanity.com/paper/1805.08440/) (2018)
- [Open Category Detection with PAC Guarantees](https://mlsanity.com/paper/1808.00529/) (2018)
- [Uncertainty-Based Out-of-Distribution Detection in Deep Reinforcement Learning](https://mlsanity.com/paper/1901.02219/) (2019)

## Built on by (most cited first, 9 of 9)

- [Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2206.01558/) (2022)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection](https://mlsanity.com/paper/2404.07099/) (2024)
- [Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation](https://mlsanity.com/paper/2503.05238/) (2025)
- [How to Enable Uncertainty Estimation in Proximal Policy Optimization](https://mlsanity.com/paper/2210.03649/) (2022)
- [Safety Monitoring for Learning-Enabled Cyber-Physical Systems in Out-of-Distribution Scenarios](https://mlsanity.com/paper/2504.13478/) (2025)
- [RAPT: Model-Predictive Out-of-Distribution Detection and Failure Diagnosis for Sim-to-Real Humanoid Deployment](https://mlsanity.com/paper/2602.01515/) (2026)
- [OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning](https://mlsanity.com/paper/2607.12523/) (2026)
- [DEEDEE: Fast and Scalable Out-of-Distribution Dynamics Detection](https://mlsanity.com/paper/2510.21638/) (2025)

## Abstract (arXiv)

Reinforcement Learning (RL) based solutions are being adopted in a variety of domains including robotics, health care and industrial automation. Most focus is given to when these solutions work well, but they fail when presented with out of distribution inputs. RL policies share the same faults as most machine learning models. Out of distribution detection for RL is generally not well covered in the literature, and there is a lack of benchmarks for this task. In this work we propose a benchmark to evaluate OOD detection methods in a Reinforcement Learning setting, by modifying the physical parameters of non-visual standard environments or corrupting the state observation for visual environments. We discuss ways to generate custom RL environments that can produce OOD data, and evaluate three uncertainty methods for the OOD detection task. Our results show that ensemble methods have the best OOD detection performance with a lower standard deviation across multiple environments.