mlsanityData to 7 Oct 2026

Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination

arXiv 2405.18556 ↗First posted 28 May 2024cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

In the rapidly changing healthcare landscape, the implementation of offline reinforcement learning (RL) in dynamic treatment regimes (DTRs) presents a mix of unprecedented opportunities and challenges. This position paper offers a critical examination of the current status of offline RL in the context of DTRs. We argue for a reassessment of applying RL in DTRs, citing concerns such as inconsistent and potentially inconclusive evaluation metrics, the absence of naive and supervised learning baselines, and the diverse choice of RL formulation in existing research. Through a case study with more than 17,000 evaluation experiments using a publicly available Sepsis dataset, we demonstrate that the performance of RL algorithms can significantly vary with changes in evaluation metrics and Markov Decision Process (MDP) formulations. Surprisingly, it is observed that in some instances, RL algorithms can be surpassed by random baselines subjected to policy evaluation methods and reward design. This calls for more careful policy evaluation and algorithm development in future DTR works. Additionally, we discussed potential enhancements toward more reliable development of RL-based dynamic treatment regimes and invited further discussion within the community. Code is available at https://github.com/GilesLuo/ReassessDTR.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 66 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

30 of 66 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications15 Feb
  2. 2022Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning5 Oct
  3. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings23 Jul

Other references · 27

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  8. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  9. 2021Offline RL Without Off-Policy Evaluationcited by 150
  10. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  11. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  12. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
15 more
  1. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  2. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  3. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  4. 2019Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learningcited by 34
  5. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  6. 2018Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Datacited by 23
  7. 2018Model-Based Reinforcement Learning for Sepsis Treatmentcited by 21
  8. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  9. 2018Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matterscited by 13
  10. 2016Accelerated Gradient Temporal Difference Learningcited by 10
  11. 2019Does the "Artificial Intelligence Clinician" learn optimal treatment strategies for sepsis in intensive care?cited by 9
  12. 2020An implicit function learning approach for parametric modal regressioncited by 7
  13. 2022Offline Policy Optimization with Eligible Actionscited by 7
  14. 2022Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Spacecited by 4
  15. 2017Effective sketching methods for value function approximationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Supportcited by 8
  2. 2026Agentifying Patient Dynamics within LLMs through Interacting with Clinical World Modelcited by 1
  3. 2025UACER: An Uncertainty-Adaptive Critic Ensemble Framework for Robust Adversarial Reinforcement Learningcited by 0
  4. 2026Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatmentcited by 0
  5. 2026Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Semi-Markov Offline Reinforcement Learning for Healthcare
  2. 2025POLAR: A Pessimistic Model-based Policy Learning Algorithm for Dynamic Treatment Regimes
  3. 2022Federated Offline Reinforcement Learning
  4. 2025Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment
  5. 2021Offline reinforcement learning with uncertainty for treatment strategies in sepsis
  6. 2026medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functions
  7. 2026Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management
  8. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment