# Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results

arXiv: https://arxiv.org/abs/2107.04982 · HTML: https://mlsanity.com/paper/2107.04982/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 11 Jul 2021.

## Builds on (most cited first, 10 of 10)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Deep Structured Energy Based Models for Anomaly Detection](https://mlsanity.com/paper/1605.07717/) (2016)
- [Anomaly Detection using One-Class Neural Networks](https://mlsanity.com/paper/1802.06360/) (2018)
- [Online Safety Assurance for Deep Reinforcement Learning](https://mlsanity.com/paper/2010.03625/) (2020)

## Built on by (most cited first, 14 of 14)

- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2206.01558/) (2022)
- [Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection](https://mlsanity.com/paper/2404.07099/) (2024)
- [Conformal Prediction Intervals for Markov Decision Process Trajectories](https://mlsanity.com/paper/2206.04860/) (2022)
- [LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty](https://mlsanity.com/paper/2209.11422/) (2022)
- [Novelty Detection in Reinforcement Learning with World Models](https://mlsanity.com/paper/2310.08731/) (2023)
- [Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation](https://mlsanity.com/paper/2503.05238/) (2025)
- [A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2109.09889/) (2021)
- [Building Real-time Awareness of Out-of-distribution in Trajectory Prediction for Autonomous Vehicles](https://mlsanity.com/paper/2409.17277/) (2024)
- [Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning](https://mlsanity.com/paper/2307.05209/) (2023)
- [Mahalanobis-Guided Latent OOD Detection for Hybrid ES-DRL Control in Time-Varying Systems](https://mlsanity.com/paper/2606.11474/) (2026)
- [OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning](https://mlsanity.com/paper/2607.12523/) (2026)
- [YRC-Bench: A Benchmark for Learning to Coordinate with Experts](https://mlsanity.com/paper/2502.09583/) (2025)
- [DEEDEE: Fast and Scalable Out-of-Distribution Dynamics Detection](https://mlsanity.com/paper/2510.21638/) (2025)

## Abstract (arXiv)

We study the problem of out-of-distribution dynamics (OODD) detection, which involves detecting when the dynamics of a temporal process change compared to the training-distribution dynamics. This is relevant to applications in control, reinforcement learning (RL), and multi-variate time-series, where changes to test time dynamics can impact the performance of learning controllers/predictors in unknown ways. This problem is particularly important in the context of deep RL, where learned controllers often overfit to the training environment. Currently, however, there is a lack of established OODD benchmarks for the types of environments commonly used in RL research. Our first contribution is to design a set of OODD benchmarks derived from common RL environments with varying types and intensities of OODD. Our second contribution is to design a strong OODD baseline approach based on recurrent implicit quantile network (RIQN), which monitors autoregressive prediction errors for OODD detection. In addition to RIQN, we introduce and test three other simpler baselines. Our final contribution is to evaluate our baseline approaches on the benchmarks to provide results for future comparison.