mlsanityData to 14 Oct 2026

Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment

arXiv 2407.19380 ↗First posted 28 Jul 2024cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → RL for clinical treatment and surgery

Abstract (arXiv)

Offline reinforcement learning has shown promise for solving tasks in safety-critical settings, such as clinical decision support. Its application, however, has been limited by the lack of interpretability and interactivity for clinicians. To address these challenges, we propose the medical decision transformer (MeDT), a novel and versatile framework based on the goal-conditioned reinforcement learning paradigm for sepsis treatment recommendation. MeDT uses the decision transformer architecture to learn a policy for drug dosage recommendation. During offline training, MeDT utilizes collected treatment trajectories to predict administered treatments for each time step, incorporating known treatment outcomes, target acuity scores, past treatment decisions, and current and past medical states. This analysis enables MeDT to capture complex dependencies among a patient's medical history, treatment decisions, outcomes, and short-term effects on stability. Our proposed conditioning uses acuity scores to address sparse reward issues and to facilitate clinician-model interactions, enhancing decision-making. Following training, MeDT can generate tailored treatment recommendations by conditioning on the desired positive outcome (survival) and user-specified short-term stability improvements. We carry out rigorous experiments on data from the MIMIC-III dataset and use off-policy evaluation to demonstrate that MeDT recommends interventions that outperform or are competitive with existing offline reinforcement learning methods while enabling a more interpretable, personalized and clinician-directed approach.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 57 references that are in the map. First posted on arXiv 28 Jul 2024.

Builds on

34 of 57 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  4. 2020Quantifying Attention Flow in Transformerscited by 775
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  6. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  7. 2022Transformers in Time Series: A Surveycited by 497
  8. 2021A Survey of Transformerscited by 401
  9. 2019Is Attention Interpretable?cited by 389
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  11. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  12. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
22 more
  1. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  2. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  3. 2019Batch Policy Learning under Constraintscited by 244
  4. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
  5. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  6. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  7. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  8. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  9. 2021A Survey on Interpretable Reinforcement Learningcited by 73
  10. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  13. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
  14. 2020Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitionscited by 41
  15. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  16. 2019Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learningcited by 34
  17. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  18. 2018Model-Based Reinforcement Learning for Sepsis Treatmentcited by 21
  19. 2022Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixelscited by 13
  20. 2019Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPscited by 12
  21. 2023Transformers in Reinforcement Learning: A Surveycited by 12
  22. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Supportcited by 9
  2. 2025Benchmarking Offline Multi-Objective Reinforcement Learning in Critical Carecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline reinforcement learning with uncertainty for treatment strategies in sepsis
  2. 2022Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
  3. 2024Reinforced Sequential Decision-Making for Sepsis Treatment: The POSNEGDM Framework with Mortality Classifier and Transformer
  4. 2024How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models
  5. 2026Attention-Based Offline Reinforcement Learning and Clustering for Interpretable Sepsis Treatment
  6. 2026Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment
  7. 2026Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation
  8. 2026Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits