mlsanityData to 14 Oct 2026

Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

arXiv 2301.05664 ↗First posted 13 Jan 2023cs.LG stat.MLcited by 6

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

In safety-critical decision-making scenarios being able to identify worst-case outcomes, or dead-ends is crucial in order to develop safe and reliable policies in practice. These situations are typically rife with uncertainty due to unknown or stochastic characteristics of the environment as well as limited offline training data. As a result, the value of a decision at any time point should be based on the distribution of its anticipated effects. We propose a framework to identify worst-case decision points, by explicitly estimating distributions of the expected return of a decision. These estimates enable earlier indication of dead-ends in a manner that is tunable based on the risk tolerance of the designed task. We demonstrate the utility of Distributional Dead-end Discovery (DistDeD) in a toy domain as well as when assessing the risk of severely ill patients in the intensive care unit reaching a point where death is unavoidable. We find that DistDeD significantly improves over prior discovery approaches, providing indications of the risk 10 hours earlier on average as well as increasing detection by 20%.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 56 references that are in the map. First posted on arXiv 13 Jan 2023.

Builds on

36 of 56 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Neural Ordinary Differential Equationscited by 3,779
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2017Constrained Policy Optimizationcited by 983
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  11. 2017Safe Reinforcement Learning via Shieldingcited by 449
  12. 2018Reward Constrained Policy Optimizationcited by 417
24 more
  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 366
  3. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  4. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  5. 2020Critic Regularized Regressioncited by 275
  6. 2022On Neural Differential Equationscited by 268
  7. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  8. 2020CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guaranteecited by 134
  9. 2013Risk-sensitive Reinforcement Learningcited by 130
  10. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  11. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  12. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  13. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 88
  14. 2021Conservative Offline Distributional Reinforcement Learningcited by 79
  15. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  16. 2022Constrained Update Projection Approach to Safe Policy Optimizationcited by 58
  17. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  18. 2021Neural Controlled Differential Equations for Online Prediction Taskscited by 46
  19. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 41
  20. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 39
  21. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  22. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  23. 2020Towards Safe Policy Improvement for Non-Stationary MDPscited by 26
  24. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 0; 2025: 1; 2026: 1.

6 papers.

  1. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  2. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  3. 2023Safe Reinforcement Learning with Dead-Ends Avoidance and Recoverycited by 5
  4. 2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
  5. 2023Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Carecited by 2
  6. 2026Chance-constrained selection of sequential intervention strategies from counterfactual estimatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  2. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  3. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  4. 2026Evaluation Metrics for Safe Reinforcement Learning
  5. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
  6. 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare
  7. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learning
  8. 2024Revisiting Safe Exploration in Safe Reinforcement learning