# Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection

arXiv: https://arxiv.org/abs/2404.07099 · HTML: https://mlsanity.com/paper/2404.07099/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Assessing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/1810.12282/) (2018)
- [High-dimensional, multiscale online changepoint detection](https://mlsanity.com/paper/2003.03668/) (2020)
- [Uncertainty-Based Out-of-Distribution Classification in Deep Reinforcement Learning](https://mlsanity.com/paper/2001.00496/) (2019)
- [Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results](https://mlsanity.com/paper/2107.04982/) (2021)
- [Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learning](https://mlsanity.com/paper/2112.02694/) (2021)
- [Stratospheric Aerosol Injection as a Deep Reinforcement Learning Problem](https://mlsanity.com/paper/1905.07366/) (2019)

## Built on by (most cited first, 17 of 17)

- [Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classification](https://mlsanity.com/paper/2502.02133/) (2025)
- [SleeperNets: Universal Backdoor Poisoning Attacks Against Reinforcement Learning Agents](https://mlsanity.com/paper/2405.20539/) (2024)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [Detecting Multi-Agent Collusion Through Multi-Agent Interpretability](https://mlsanity.com/paper/2604.01151/) (2026)
- [Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation](https://mlsanity.com/paper/2503.05238/) (2025)
- [Out-of-Distribution Detection for Neurosymbolic Autonomous Cyber Agents](https://mlsanity.com/paper/2412.02875/) (2024)
- [When to Trust Your Data: Enhancing Dyna-Style Model-Based Reinforcement Learning With Data Filter](https://mlsanity.com/paper/2410.12160/) (2024)
- [Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systems](https://mlsanity.com/paper/2510.21254/) (2025)
- [Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning](https://mlsanity.com/paper/2602.05089/) (2026)
- [Harnessing Feature Resonance under Arbitrary Target Alignment for Out-of-Distribution Node Detection](https://mlsanity.com/paper/2502.16076/) (2025)
- [RAPT: Model-Predictive Out-of-Distribution Detection and Failure Diagnosis for Sim-to-Real Humanoid Deployment](https://mlsanity.com/paper/2602.01515/) (2026)
- [Mahalanobis-Guided Latent OOD Detection for Hybrid ES-DRL Control in Time-Varying Systems](https://mlsanity.com/paper/2606.11474/) (2026)
- [OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning](https://mlsanity.com/paper/2607.12523/) (2026)
- [Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptation](https://mlsanity.com/paper/2609.10954/) (2026)
- [YRC-Bench: A Benchmark for Learning to Coordinate with Experts](https://mlsanity.com/paper/2502.09583/) (2025)
- [DEEDEE: Fast and Scalable Out-of-Distribution Dynamics Detection](https://mlsanity.com/paper/2510.21638/) (2025)
- [Detecting is Easy, Adapting is Hard: Local Expert Growth for Visual Model-Based Reinforcement Learning under Distribution Shift](https://mlsanity.com/paper/2604.27411/) (2026)

## Abstract (arXiv)

While reinforcement learning (RL) algorithms have been successfully applied across numerous sequential decision-making problems, their generalization to unforeseen testing environments remains a significant concern. In this paper, we study the problem of out-of-distribution (OOD) detection in RL, which focuses on identifying situations at test time that RL agents have not encountered in their training environments. We first propose a clarification of terminology for OOD detection in RL, which aligns it with the literature from other machine learning domains. We then present new benchmark scenarios for OOD detection, which introduce anomalies with temporal autocorrelation into different components of the agent-environment loop. We argue that such scenarios have been understudied in the current literature, despite their relevance to real-world situations. Confirming our theoretical predictions, our experimental results suggest that state-of-the-art OOD detectors are not able to identify such anomalies. To address this problem, we propose a novel method for OOD detection, which we call DEXTER (Detection via Extraction of Time Series Representations). By treating environment observations as time series data, DEXTER extracts salient time series features, and then leverages an ensemble of isolation forest algorithms to detect anomalies. We find that DEXTER can reliably identify anomalies across benchmark scenarios, exhibiting superior performance compared to both state-of-the-art OOD detectors and high-dimensional changepoint detectors adopted from statistics.