# Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation

arXiv: https://arxiv.org/abs/2003.06474 · HTML: https://mlsanity.com/paper/2003.06474/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Mar 2020.

## Built on by (most cited first, 8 of 8)

- [Enabling risk-aware Reinforcement Learning for medical interventions through uncertainty decomposition](https://mlsanity.com/paper/2109.07827/) (2021)
- [Learning Under Adversarial and Interventional Shifts](https://mlsanity.com/paper/2103.15933/) (2021)
- [Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes](https://mlsanity.com/paper/2407.00364/) (2024)
- [Off-Policy Evaluation Under Nonignorable Missing Data](https://mlsanity.com/paper/2507.06961/) (2025)
- [EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins](https://mlsanity.com/paper/2607.08793/) (2026)
- [TECM*: A Data-Driven Assessment to Reinforcement Learning Methods and Application to Heparin Treatment Strategy for Surgical Sepsis](https://mlsanity.com/paper/2512.10973/) (2025)
- [Offline Bayesian Aleatoric and Epistemic Uncertainty Quantification and Posterior Value Optimisation in Finite-State MDPs](https://mlsanity.com/paper/2406.02456/) (2024)
- [CNA: An AI-Oriented Comprehensive Normalized Assessment for Healthy Status and Application to Optimize RRT Strategies by Reinforcement Learning](https://mlsanity.com/paper/2609.22232/) (2026)

## Abstract (arXiv)

Our aim is to establish a framework where reinforcement learning (RL) of optimizing interventions retrospectively allows us a regulatory compliant pathway to prospective clinical testing of the learned policies in a clinical deployment. We focus on infections in intensive care units which are one of the major causes of death and difficult to treat because of the complex and opaque patient dynamics, and the clinically debated, highly-divergent set of intervention policies required by each individual patient, yet intensive care units are naturally data rich. In our work, we build on RL approaches in healthcare ("AI Clinicians"), and learn off-policy continuous dosing policy of pharmaceuticals for sepsis treatment using historical intensive care data under partially observable MDPs (POMDPs). POMPDs capture uncertainty in patient state better by taking in all historical information, yielding an efficient representation, which we investigate through ablations. We compensate for the lack of exploration in our retrospective data by evaluating each encountered state with a best-first tree search. We mitigate state distributional shift by optimizing our policy in the vicinity of the clinicians' compound policy. Crucially, we evaluate our model recommendations using not only conventional policy evaluations but a novel framework that incorporates human experts: a model-agnostic pre-clinical evaluation method to estimate the accuracy and uncertainty of clinician's decisions versus our system recommendations when confronted with the same individual patient history ("shadow mode").