mlsanityData to 7 Oct 2026

Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications

arXiv 2302.07549 ↗First posted 15 Feb 2023cs.LGcited by 17

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

There is increasing interest in data-driven approaches for recommending optimal treatment strategies in many chronic disease management and critical care applications. Reinforcement learning methods are well-suited to this sequential decision-making problem, but must be trained and evaluated exclusively on retrospective medical record datasets as direct online exploration is unsafe and infeasible. Despite this requirement, the vast majority of treatment optimization studies use off-policy RL methods (e.g., Double Deep Q Networks (DDQN) or its variants) that are known to perform poorly in purely offline settings. Recent advances in offline RL, such as Conservative Q-Learning (CQL), offer a suitable alternative. But there remain challenges in adapting these approaches to real-world applications where suboptimal examples dominate the retrospective dataset and strict safety constraints need to be satisfied. In this work, we introduce a practical and theoretically grounded transition sampling approach to address action imbalance during offline RL training. We perform extensive experiments on two real-world tasks for diabetes and sepsis treatment optimization to compare performance of the proposed approach against prominent off-policy and offline RL baselines (DDQN and CQL). Across a range of principled and clinically relevant metrics, we show that our proposed approach enables substantial improvements in expected health outcomes and in accordance with relevant practice and safety guidelines.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 66 references that are in the map. First posted on arXiv 15 Feb 2023.

Builds on

28 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Semi-Markov Offline Reinforcement Learning for Healthcare17 Mar

Other references · 27

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  8. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  10. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  12. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
15 more
  1. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  2. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  3. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  4. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  5. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  6. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  7. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  8. 2017OptLayer - Practical Constrained Optimization for Deep Reinforcement Learning in the Real Worldcited by 64
  9. 2019Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learningcited by 34
  10. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  11. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  12. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  13. 2020Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validationcited by 18
  14. 2022Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learningcited by 9
  15. 2020Controlling Level of Unconsciousness by Titrating Propofol with Deep Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 8; 2026: 3.

17 papers.

Sort:

By area:

  1. 2025medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Supportcited by 8
  2. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learningcited by 7
  3. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  4. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexaminationcited by 5
  5. 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
  6. 2025Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Datasetcited by 4
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learningcited by 4
  8. 2025Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategiescited by 4
  9. 2023End-to-end Offline Reinforcement Learning for Glycemia Controlcited by 2
  10. 2025MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representationcited by 2
  11. 2024Development and Validation of Heparin Dosing Policies Using an Offline Reinforcement Learning Algorithmcited by 1
  12. 2025Reinforcement Learning for Accelerator Beamline Control: a simulation-based approachcited by 1
5 more
  1. 2025PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learningcited by 1
  2. 2025ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orderscited by 0
  3. 2026RL-ABC: Reinforcement Learning for Accelerator Beamline Controlcited by 0
  4. 2026Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatmentcited by 0
  5. 2026Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
  2. 2022A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  4. 2023Offline Reinforcement Learning with Imbalanced Datasets
  5. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes
  6. 2022Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
  7. 2021The Least Restriction for Offline Reinforcement Learning
  8. 2022Federated Offline Reinforcement Learning