# An Empirical Study of Representation Learning for Reinforcement Learning in Healthcare

arXiv: https://arxiv.org/abs/2011.11235 · HTML: https://mlsanity.com/paper/2011.11235/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 30 of 34)

- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [Conservative Data Sharing for Multi-Task Offline Reinforcement Learning](https://mlsanity.com/paper/2109.08128/) (2021)
- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [Neighborhood Contrastive Learning Applied to Online Patient Monitoring](https://mlsanity.com/paper/2106.05142/) (2021)
- [Medical Dead-ends and Learning to Identify High-risk States and Treatments](https://mlsanity.com/paper/2110.04186/) (2021)
- [Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications](https://mlsanity.com/paper/2302.07549/) (2023)
- [Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment](https://mlsanity.com/paper/2101.08477/) (2021)
- [Offline Multitask Representation Learning for Reinforcement Learning](https://mlsanity.com/paper/2403.11574/) (2024)
- [Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders](https://mlsanity.com/paper/2302.00662/) (2023)
- [Finding Counterfactually Optimal Action Sequences in Continuous State Spaces](https://mlsanity.com/paper/2306.03929/) (2023)
- [Temporal Supervised Contrastive Learning for Modeling Patient Risk Progression](https://mlsanity.com/paper/2312.05933/) (2023)
- [medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Support](https://mlsanity.com/paper/2505.19785/) (2025)
- [Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning](https://mlsanity.com/paper/2301.05664/) (2023)
- [Approximate Information States for Worst-Case Control and Learning in Uncertain Systems](https://mlsanity.com/paper/2301.05089/) (2023)
- [Enabling A Network AI Gym for Autonomous Cyber Agents](https://mlsanity.com/paper/2304.01366/) (2023)
- [Offline Reinforcement Learning with Domain-Unlabeled Data](https://mlsanity.com/paper/2404.07465/) (2024)
- [Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI](https://mlsanity.com/paper/2508.21101/) (2025)
- [How Consistent are Clinicians? Evaluating the Predictability of Sepsis Disease Progression with Dynamics Models](https://mlsanity.com/paper/2404.07148/) (2024)
- [Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment](https://mlsanity.com/paper/2407.19380/) (2024)
- [OCMDP: Observation-Constrained Markov Decision Process](https://mlsanity.com/paper/2411.07087/) (2024)
- [Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2507.06628/) (2025)
- [MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation](https://mlsanity.com/paper/2508.07681/) (2025)
- [MIMIC-Sepsis: A Curated Benchmark for Modeling and Learning from Sepsis Trajectories in the ICU](https://mlsanity.com/paper/2510.24500/) (2025)
- [The hidden risks of temporal resampling in clinical reinforcement learning](https://mlsanity.com/paper/2602.06603/) (2026)
- [Deep Normed Embeddings for Patient Representation](https://mlsanity.com/paper/2204.05477/) (2022)
- [A Framework for Empowering Reinforcement Learning Agents with Causal Analysis: Enhancing Automated Cryptocurrency Trading](https://mlsanity.com/paper/2310.09462/) (2023)
- [Development and Validation of Heparin Dosing Policies Using an Offline Reinforcement Learning Algorithm](https://mlsanity.com/paper/2409.15753/) (2024)
- [Identifying Differential Patient Care Through Inverse Intent Inference](https://mlsanity.com/paper/2411.07372/) (2024)
- [Reinforcement Learning For Survival, A Clinically Motivated Method For Critically Ill Patients](https://mlsanity.com/paper/2207.08040/) (2022)
- [Q-function Decomposition with Intervention Semantics with Factored Action Spaces](https://mlsanity.com/paper/2504.21326/) (2025)

## Abstract (arXiv)

Reinforcement Learning (RL) has recently been applied to sequential estimation and prediction problems identifying and developing hypothetical treatment strategies for septic patients, with a particular focus on offline learning with observational data. In practice, successful RL relies on informative latent states derived from sequential observations to develop optimal treatment strategies. To date, how best to construct such states in a healthcare setting is an open question. In this paper, we perform an empirical study of several information encoding architectures using data from septic patients in the MIMIC-III dataset to form representations of a patient state. We evaluate the impact of representation dimension, correlations with established acuity scores, and the treatment policies derived from them. We find that sequentially formed state representations facilitate effective policy learning in batch settings, validating a more thoughtful approach to representation learning that remains faithful to the sequential and partial nature of healthcare data.