mlsanityData to 14 Oct 2026

Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation

arXiv 2503.05238 ↗First posted 7 Mar 2025cs.LGcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

An issue concerning the use of deep reinforcement learning (RL) agents is whether they can be trusted to perform reliably when deployed, as training environments may not reflect real-life environments. Anticipating instances outside their training scope, learning-enabled systems are often equipped with out-of-distribution (OOD) detectors that alert when a trained system encounters a state it does not recognize or in which it exhibits uncertainty. There exists limited work conducted on the problem of OOD detection within RL, with prior studies being unable to achieve a consensus on the definition of OOD execution within the context of RL. By framing our problem using a Markov Decision Process, we assume there is a transition distribution mapping each state-action pair to another state with some probability. Based on this, we consider the following definition of OOD execution within RL: A transition is OOD if its probability during real-life deployment differs from the transition distribution encountered during training. As such, we utilize conditional variational autoencoders (CVAE) to approximate the transition dynamics of the training environment and implement a conformity-based detector using reconstruction loss that is able to guarantee OOD detection with a pre-determined confidence level. We evaluate our detector by adapting existing benchmarks and compare it with existing OOD detection models for RL.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 38 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

25 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection10 Apr

Other references · 24

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2016OpenAI Gymcited by 1,736
  4. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
  5. 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
  6. 2018Classification-Reconstruction Learning for Open-Set Recognitioncited by 146
  7. 2020Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encodercited by 145
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
  9. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  10. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 88
  11. 2021Conservative Offline Distributional Reinforcement Learningcited by 79
  12. 2018Improving Reconstruction Autoencoder Out-of-distribution Detection with Mahalanobis Distancecited by 74
12 more
  1. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 72
  2. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
  3. 2020Real-time Out-of-distribution Detection in Learning-Enabled Cyber-Physical Systemscited by 46
  4. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
  5. 2022Sample Efficient Deep Reinforcement Learning via Uncertainty Estimationcited by 27
  6. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  7. 2019Uncertainty-Based Out-of-Distribution Classification in Deep Reinforcement Learningcited by 15
  8. 2021Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Resultscited by 14
  9. 2020Policy Entropy for Out-of-Distribution Classificationcited by 11
  10. 2021Benchmark for Out-of-Distribution Detection in Deep Reinforcement Learningcited by 9
  11. 2022Out of Distribution Reasoning by Weakly-Supervised Disentangled Logic Variational Autoencodercited by 2
  12. 2023PAC-Based Formal Verification for Out-of-Distribution Data Detectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3
  2. 2025Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systemscited by 2
  3. 2026RAPT: Model-Predictive Out-of-Distribution Detection and Failure Diagnosis for Sim-to-Real Humanoid Deploymentcited by 1
  4. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RLcited by 0
  5. 2025DEEDEE: Fast and Scalable Out-of-Distribution Dynamics Detectioncited by 0
  6. 2026Think Fast, Plan Selectively: Adaptive Deliberation for Efficient Data-Driven MPCcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OOD-RL-Bench: A Benchmark Framework for Out-of-Distribution Detection in Reinforcement Learning
  2. 2026Theoretical Grounding of Out-Of-Distribution Detection With Reinforcement Learning Optimizer
  3. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
  4. 2025Variational OOD State Correction for Offline Reinforcement Learning
  5. 2026Generative OOD-regularized Model-based Policy Optimization
  6. 2026Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning
  7. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  8. 2026Smaller Abstract State Spaces Enable Cross-Scale Generalization in Reinforcement Learning