# Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications arXiv: https://arxiv.org/abs/2302.07549 ยท HTML: https://mlsanity.com/paper/2302.07549/ ## Key facts - Cited by 17 papers in the map (data through 14 Oct 2026). - Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. - Builds on 28 of its 66 references that are in the map. - First posted on arXiv 15 Feb 2023. ## Placed in - Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/) ## Builds on (most cited first, 28 of 28) - [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013) - [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020) - [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015) - [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018) - [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017) - [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019) - [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019) - [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020) - [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021) - [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019) - [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021) - [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021) - [Mildly Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.04745/) (2022) - [BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning](https://mlsanity.com/paper/1910.12179/) (2019) - [Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.03400/) (2021) - [Diagnosing Bottlenecks in Deep Q-learning Algorithms](https://mlsanity.com/paper/1902.10250/) (2019) - [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017) - [COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation](https://mlsanity.com/paper/2204.08957/) (2022) - [OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real World](https://mlsanity.com/paper/1709.07643/) (2017) - [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017) - [Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning](https://mlsanity.com/paper/1901.04670/) (2019) - [An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2011.11235/) (2020) - [Medical Dead-ends and Learning to Identify High-risk States and Treatments](https://mlsanity.com/paper/2110.04186/) (2021) - [Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policies](https://mlsanity.com/paper/2007.12678/) (2020) - [Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation](https://mlsanity.com/paper/2005.09059/) (2020) - [Semi-Markov Offline Reinforcement Learning for Healthcare](https://mlsanity.com/paper/2203.09365/) (2022) - [Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2208.12433/) (2022) - [Controlling Level of Unconsciousness by Titrating Propofol with Deep Reinforcement Learning](https://mlsanity.com/paper/2008.12333/) (2020) ## Built on by (most cited first, 17 of 17) - [medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Support](https://mlsanity.com/paper/2505.19785/) (2025) - [SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning](https://mlsanity.com/paper/2402.13147/) (2024) - [TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents](https://mlsanity.com/paper/2404.12530/) (2024) - [Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination](https://mlsanity.com/paper/2405.18556/) (2024) - [Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare](https://mlsanity.com/paper/2410.07525/) (2024) - [Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset](https://mlsanity.com/paper/2502.18955/) (2025) - [Evaluation-Time Policy Switching for Offline Reinforcement Learning](https://mlsanity.com/paper/2503.12222/) (2025) - [Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies](https://mlsanity.com/paper/2505.16242/) (2025) - [End-to-end Offline Reinforcement Learning for Glycemia Control](https://mlsanity.com/paper/2310.10312/) (2023) - [MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation](https://mlsanity.com/paper/2508.07681/) (2025) - [Development and Validation of Heparin Dosing Policies Using an Offline Reinforcement Learning Algorithm](https://mlsanity.com/paper/2409.15753/) (2024) - [ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orders](https://mlsanity.com/paper/2504.17277/) (2025) - [Reinforcement Learning for Accelerator Beamline Control: a simulation-based approach](https://mlsanity.com/paper/2510.26805/) (2025) - [PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning](https://mlsanity.com/paper/2512.07342/) (2025) - [RL-ABC: Reinforcement Learning for Accelerator Beamline Control](https://mlsanity.com/paper/2604.19146/) (2026) - [Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatment](https://mlsanity.com/paper/2606.01051/) (2026) - [Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry](https://mlsanity.com/paper/2608.30442/) (2026) ## Abstract (arXiv) There is increasing interest in data-driven approaches for recommending optimal treatment strategies in many chronic disease management and critical care applications. Reinforcement learning methods are well-suited to this sequential decision-making problem, but must be trained and evaluated exclusively on retrospective medical record datasets as direct online exploration is unsafe and infeasible. Despite this requirement, the vast majority of treatment optimization studies use off-policy RL methods (e.g., Double Deep Q Networks (DDQN) or its variants) that are known to perform poorly in purely offline settings. Recent advances in offline RL, such as Conservative Q-Learning (CQL), offer a suitable alternative. But there remain challenges in adapting these approaches to real-world applications where suboptimal examples dominate the retrospective dataset and strict safety constraints need to be satisfied. In this work, we introduce a practical and theoretically grounded transition sampling approach to address action imbalance during offline RL training. We perform extensive experiments on two real-world tasks for diabetes and sepsis treatment optimization to compare performance of the proposed approach against prominent off-policy and offline RL baselines (DDQN and CQL). Across a range of principled and clinically relevant metrics, we show that our proposed approach enables substantial improvements in expected health outcomes and in accordance with relevant practice and safety guidelines.