# ICU-Sepsis: A Benchmark MDP Built from Real Medical Data

arXiv: https://arxiv.org/abs/2406.05646 · HTML: https://mlsanity.com/paper/2406.05646/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in RL for clinical treatment and surgery, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 40 references that are in the map.
- First posted on arXiv 9 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [RL for clinical treatment and surgery](https://mlsanity.com/deeprl/rl-for-clinical-treatment-and-surgery/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models](https://mlsanity.com/paper/1905.05824/) (2019)
- [Deep Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1711.09602/) (2017)
- [Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment](https://mlsanity.com/paper/2101.08477/) (2021)
- [Identifying Distinct, Effective Treatments for Acute Hypotension with SODA-RL: Safely Optimized Diverse Accurate Reinforcement Learning](https://mlsanity.com/paper/2001.03224/) (2020)

## Built on by (most cited first, 6 of 6)

- [Gymnasium: A Standard Interface for Reinforcement Learning Environments](https://mlsanity.com/paper/2407.17032/) (2024)
- [Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/2511.20913/) (2025)
- [Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MC](https://mlsanity.com/paper/2602.14505/) (2026)
- [MDPs with a State Sensing Cost](https://mlsanity.com/paper/2505.03280/) (2025)
- [Bayesian Inverse Transition Learning: Learning Dynamics From Near-Optimal Trajectories](https://mlsanity.com/paper/2411.05174/) (2024)
- [A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning](https://mlsanity.com/paper/2606.01028/) (2026)

## Abstract (arXiv)

We present ICU-Sepsis, an environment that can be used in benchmarks for evaluating reinforcement learning (RL) algorithms. Sepsis management is a complex task that has been an important topic in applied RL research in recent years. Therefore, MDPs that model sepsis management can serve as part of a benchmark to evaluate RL algorithms on a challenging real-world problem. However, creating usable MDPs that simulate sepsis care in the ICU remains a challenge due to the complexities involved in acquiring and processing patient data. ICU-Sepsis is a lightweight environment that models personalized care of sepsis patients in the ICU. The environment is a tabular MDP that is widely compatible and is challenging even for state-of-the-art RL algorithms, making it a valuable tool for benchmarking their performance. However, we emphasize that while ICU-Sepsis provides a standardized environment for evaluating RL algorithms, it should not be used to draw conclusions that guide medical practice.