Benchmarking Offline Multi-Objective Reinforcement Learning in Critical Care
On the map: Offline RL and imitation learning from data → Safe offline RL
Abstract (arXiv)
In critical care settings such as the Intensive Care Unit, clinicians face the complex challenge of balancing conflicting objectives, primarily maximizing patient survival while minimizing resource utilization (e.g., length of stay). Single-objective Reinforcement Learning approaches typically address this by optimizing a fixed scalarized reward function, resulting in rigid policies that fail to adapt to varying clinical priorities. Multi-objective Reinforcement Learning (MORL) offers a solution by learning a set of optimal policies along the Pareto Frontier, allowing for dynamic preference selection at test time. However, applying MORL in healthcare necessitates strict offline learning from historical data. In this paper, we benchmark three offline MORL algorithms, Conditioned Conservative Pareto Q-Learning (CPQL), Adaptive CPQL, and a modified Pareto Efficient Decision Agent (PEDA) Decision Transformer (PEDA DT), against three scalarized single-objective baselines (BC, CQL, and DDQN) on the MIMIC-IV dataset. Using Off-Policy Evaluation (OPE) metrics, we demonstrate that PEDA DT algorithm offers superior flexibility compared to static scalarized baselines. Notably, our results extend previous findings on single-objective Decision Transformers in healthcare, confirming that sequence modeling architectures remain robust and effective when scaled to multi-objective conditioned generation. These findings suggest that offline MORL is a promising framework for enabling personalized, adjustable decision-making in critical care without the need for retraining.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 12 of its 18 references that are in the map. First posted on arXiv 8 Dec 2025.
Builds on
12 of 18 references.
References · 12
- 2017Attention Is All You Needcited by 59,932
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2018World Modelscited by 1,444
- 2019Batch Policy Learning under Constraintscited by 242
- 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
- 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
- 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
- 2023Explaining Reinforcement Learning with Shapley Valuescited by 26
- 2025medDreamer: Model-Based Reinforcement Learning with Latent Imagination on Complex EHRs for Clinical Decision Supportcited by 8
- 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatmentcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care
- 2026Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation
- 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
- 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
- 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare
- 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
- 2022A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies
- 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings