# Medical Dead-ends and Learning to Identify High-risk States and Treatments

arXiv: https://arxiv.org/abs/2110.04186 · HTML: https://mlsanity.com/paper/2110.04186/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 23 of 23)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog](https://mlsanity.com/paper/1907.00456/) (2019)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning](https://mlsanity.com/paper/2103.06326/) (2021)
- [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017)
- [Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning](https://mlsanity.com/paper/1901.04670/) (2019)
- [An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2011.11235/) (2020)
- [Off-Policy Evaluation via Off-Policy Classification](https://mlsanity.com/paper/1906.01624/) (2019)
- [Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policies](https://mlsanity.com/paper/2007.12678/) (2020)
- [An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection](https://mlsanity.com/paper/1708.05894/) (2017)
- [On overfitting and asymptotic bias in batch reinforcement learning with partial observability](https://mlsanity.com/paper/1709.07796/) (2017)
- [Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs](https://mlsanity.com/paper/1905.07465/) (2019)

## Built on by (most cited first, 28 of 28)

- [Concise Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2504.05185/) (2025)
- [The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2305.16589/) (2023)
- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path](https://mlsanity.com/paper/2206.02678/) (2022)
- [Semi-Markov Offline Reinforcement Learning for Healthcare](https://mlsanity.com/paper/2203.09365/) (2022)
- [Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications](https://mlsanity.com/paper/2302.07549/) (2023)
- [Systematic Rectification of Language Models via Dead-end Analysis](https://mlsanity.com/paper/2302.14003/) (2023)
- [Learning General World Models in a Handful of Reward-Free Deployments](https://mlsanity.com/paper/2210.12719/) (2022)
- [Quasi-optimal Reinforcement Learning with Continuous Actions](https://mlsanity.com/paper/2301.08940/) (2023)
- [Connected and Automated Vehicles in Mixed-Traffic: Learning Human Driver Behavior for Effective On-Ramp Merging](https://mlsanity.com/paper/2304.00397/) (2023)
- [TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents](https://mlsanity.com/paper/2404.12530/) (2024)
- [Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning](https://mlsanity.com/paper/2301.05664/) (2023)
- [The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learning](https://mlsanity.com/paper/2506.13672/) (2025)
- [Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery](https://mlsanity.com/paper/2306.13944/) (2023)
- [Minimum Levels of Interpretability for Artificial Moral Agents](https://mlsanity.com/paper/2307.00660/) (2023)
- [Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare](https://mlsanity.com/paper/2410.07525/) (2024)
- [Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/2511.20913/) (2025)
- [Approximate Information States for Worst-Case Control and Learning in Uncertain Systems](https://mlsanity.com/paper/2301.05089/) (2023)
- [Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care](https://mlsanity.com/paper/2306.08044/) (2023)
- [Successor Features for Efficient Multisubject Controlled Text Generation](https://mlsanity.com/paper/2311.04921/) (2023)
- [The hidden risks of temporal resampling in clinical reinforcement learning](https://mlsanity.com/paper/2602.06603/) (2026)
- [PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning](https://mlsanity.com/paper/2512.07342/) (2025)
- [Extracting Expert's Goals by What-if Interpretable Modeling](https://mlsanity.com/paper/2110.15165/) (2021)
- [Learned Risk Metric Maps for Kinodynamic Systems](https://mlsanity.com/paper/2302.14803/) (2023)
- [Learning and DiSentangling Patient Static Information from Time-series Electronic HEalth Record (STEER)](https://mlsanity.com/paper/2309.11373/) (2023)
- [A Dynamical View of the Question of Why](https://mlsanity.com/paper/2402.10240/) (2024)
- [Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories](https://mlsanity.com/paper/2411.05174/) (2024)
- [Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learning](https://mlsanity.com/paper/2606.19481/) (2026)

## Abstract (arXiv)

Machine learning has successfully framed many sequential decision making problems as either supervised prediction, or optimal decision-making policy identification via reinforcement learning. In data-constrained offline settings, both approaches may fail as they assume fully optimal behavior or rely on exploring alternatives that may not exist. We introduce an inherently different approach that identifies possible "dead-ends" of a state space. We focus on the condition of patients in the intensive care unit, where a "medical dead-end" indicates that a patient will expire, regardless of all potential future treatment sequences. We postulate "treatment security" as avoiding treatments with probability proportional to their chance of leading to dead-ends, present a formal proof, and frame discovery as an RL problem. We then train three independent deep neural models for automated state construction, dead-end discovery and confirmation. Our empirical results discover that dead-ends exist in real clinical data among septic patients, and further reveal gaps between secure treatments and those that were administered.