# Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment

arXiv: https://arxiv.org/abs/2101.08477 · HTML: https://mlsanity.com/paper/2101.08477/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 47 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [An Analysis of Categorical Distributional Reinforcement Learning](https://mlsanity.com/paper/1802.08163/) (2018)
- [Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.07971/) (2020)
- [Evaluating Reinforcement Learning Algorithms in Observational Health Settings](https://mlsanity.com/paper/1805.12298/) (2018)
- [Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning](https://mlsanity.com/paper/1901.04670/) (2019)
- [Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms](https://mlsanity.com/paper/2004.10710/) (2020)
- [An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2011.11235/) (2020)
- [Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs](https://mlsanity.com/paper/1905.07465/) (2019)

## Built on by (most cited first, 16 of 16)

- [Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption](https://mlsanity.com/paper/2402.08991/) (2024)
- [ICU-Sepsis: A Benchmark MDP Built from Real Medical Data](https://mlsanity.com/paper/2406.05646/) (2024)
- [Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data](https://mlsanity.com/paper/2505.09496/) (2025)
- [Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MC](https://mlsanity.com/paper/2602.14505/) (2026)
- [Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care](https://mlsanity.com/paper/2306.08044/) (2023)
- [How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models](https://mlsanity.com/paper/2404.07148/) (2024)
- [The hidden risks of temporal resampling in clinical reinforcement learning](https://mlsanity.com/paper/2602.06603/) (2026)
- [EHR-MPC: Inference-Time Control for Sepsis Treatment with Generative Patient Digital Twins](https://mlsanity.com/paper/2607.08793/) (2026)
- [Deep Normed Embeddings for Patient Representation](https://mlsanity.com/paper/2204.05477/) (2022)
- [Methodology for Interpretable Reinforcement Learning for Optimizing Mechanical Ventilation](https://mlsanity.com/paper/2404.03105/) (2024)
- [Reinforcement Learning For Survival, A Clinically Motivated Method For Critically Ill Patients](https://mlsanity.com/paper/2207.08040/) (2022)
- [MiranDa: Mimicking the Learning Processes of Human Doctors to Achieve Causal Inference for Medication Recommendation](https://mlsanity.com/paper/2408.01445/) (2024)
- [Learning optimal treatment strategies for intraoperative hypotension using deep reinforcement learning](https://mlsanity.com/paper/2505.21596/) (2025)
- [Intelligent Reasoning Cues: A Framework and Case Study of the Roles of AI Information in Complex Decisions](https://mlsanity.com/paper/2602.00259/) (2026)
- [Modeling Clinical Concern Trajectories in Language Model Agents](https://mlsanity.com/paper/2604.27872/) (2026)
- [Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Models](https://mlsanity.com/paper/2606.04562/) (2026)

## Abstract (arXiv)

Sepsis is a potentially life threatening inflammatory response to infection or severe tissue damage. It has a highly variable clinical course, requiring constant monitoring of the patient's state to guide the management of intravenous fluids and vasopressors, among other interventions. Despite decades of research, there's still debate among experts on optimal treatment. Here, we combine for the first time, distributional deep reinforcement learning with mechanistic physiological models to find personalized sepsis treatment strategies. Our method handles partial observability by leveraging known cardiovascular physiology, introducing a novel physiology-driven recurrent autoencoder, and quantifies the uncertainty of its own results. Moreover, we introduce a framework for uncertainty aware decision support with humans in the loop. We show that our method learns physiologically explainable, robust policies that are consistent with clinical knowledge. Further our method consistently identifies high risk states that lead to death, which could potentially benefit from more frequent vasopressor administration, providing valuable guidance for future research