# Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPs

arXiv: https://arxiv.org/abs/1905.07465 · HTML: https://mlsanity.com/paper/1905.07465/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2019.

## Built on by (most cited first, 12 of 12)

- [Medical Dead-ends and Learning to Identify High-risk States and Treatments](https://mlsanity.com/paper/2110.04186/) (2021)
- [Semi-Markov Offline Reinforcement Learning for Healthcare](https://mlsanity.com/paper/2203.09365/) (2022)
- [Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment](https://mlsanity.com/paper/2101.08477/) (2021)
- [Offline reinforcement learning with uncertainty for treatment strategies in sepsis](https://mlsanity.com/paper/2107.04491/) (2021)
- [Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning](https://mlsanity.com/paper/2206.14198/) (2022)
- [Medical Knowledge Integration into Reinforcement Learning Algorithms for Dynamic Treatment Regimes](https://mlsanity.com/paper/2407.00364/) (2024)
- [Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment](https://mlsanity.com/paper/2407.19380/) (2024)
- [Optimal Control of Mechanical Ventilators with Learned Respiratory Dynamics](https://mlsanity.com/paper/2411.07971/) (2024)
- [Deep Normed Embeddings for Patient Representation](https://mlsanity.com/paper/2204.05477/) (2022)
- [Reinforcement Learning For Survival, A Clinically Motivated Method For Critically Ill Patients](https://mlsanity.com/paper/2207.08040/) (2022)
- [Deep Attention Q-Network for Personalized Treatment Recommendation](https://mlsanity.com/paper/2307.01519/) (2023)
- [Exploring a Graph-based Approach to Offline Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/2509.03393/) (2025)

## Abstract (arXiv)

Health-related data is noisy and stochastic in implying the true physiological states of patients, limiting information contained in single-moment observations for sequential clinical decision making. We model patient-clinician interactions as partially observable Markov decision processes (POMDPs) and optimize sequential treatment based on belief states inferred from history sequence. To facilitate inference, we build a variational generative model and boost state representation with a recurrent neural network (RNN), incorporating an auxiliary loss from sequence auto-encoding. Meanwhile, we optimize a continuous policy of drug levels with an actor-critic method where policy gradients are obtained from a stablized off-policy estimate of advantage function, with the value of belief state backed up by parallel best-first suffix trees. We exploit our methodology in optimizing dosages of vasopressor and intravenous fluid for sepsis patients using a retrospective intensive care dataset and evaluate the learned policy with off-policy policy evaluation (OPPE). The results demonstrate that modelling as POMDPs yields better performance than MDPs, and that incorporating heuristic search improves sample efficiency.