# Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategies

arXiv: https://arxiv.org/abs/2505.16242 · HTML: https://mlsanity.com/paper/2505.16242/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 25 of its 57 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [Safe Reinforcement Learning in Constrained Markov Decision Processes](https://mlsanity.com/paper/2008.06626/) (2020)
- [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [A Survey of Constraint Formulations in Safe Reinforcement Learning](https://mlsanity.com/paper/2402.02025/) (2024)
- [Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms](https://mlsanity.com/paper/2310.03225/) (2023)
- [Supported Trust Region Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2311.08935/) (2023)
- [Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications](https://mlsanity.com/paper/2302.07549/) (2023)
- [Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression](https://mlsanity.com/paper/2410.19400/) (2024)
- [Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation](https://mlsanity.com/paper/2411.09891/) (2024)
- [Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2405.13863/) (2024)
- [Robust and Kernelized Data-Enabled Predictive Control for Nonlinear Systems](https://mlsanity.com/paper/2206.01866/) (2022)
- [Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studies](https://mlsanity.com/paper/2010.04279/) (2020)
- [Flipping-based Policy for Chance-Constrained Markov Decision Processes](https://mlsanity.com/paper/2410.06474/) (2024)
- [Probabilistic reachable sets of stochastic nonlinear systems with contextual uncertainties](https://mlsanity.com/paper/2403.12379/) (2024)
- [Nonnegative forms with sublevel sets of minimal volume](https://mlsanity.com/paper/1909.10935/) (2019)

## Built on by (most cited first, 4 of 4)

- [Guardian-regularized Safe Offline Reinforcement Learning for Smart Weaning of Mechanical Circulatory Devices](https://mlsanity.com/paper/2511.06111/) (2025)
- [Generative OOD-regularized Model-based Policy Optimization](https://mlsanity.com/paper/2605.24405/) (2026)
- [A Unified Benchmark for Dynamic Medical Treatment Reinforcement Learning](https://mlsanity.com/paper/2606.01028/) (2026)
- [Interaction-Limited Safe Continuous-Time RL for Dynamical Medical Treatment](https://mlsanity.com/paper/2606.01051/) (2026)

## Abstract (arXiv)

When applying offline reinforcement learning (RL) in healthcare scenarios, the out-of-distribution (OOD) issues pose significant risks, as inappropriate generalization beyond clinical expertise can result in potentially harmful recommendations. While existing methods like conservative Q-learning (CQL) attempt to address the OOD issue, their effectiveness is limited by only constraining action selection by suppressing uncertain actions. This action-only regularization imitates clinician actions that prioritize short-term rewards, but it fails to regulate downstream state trajectories, thereby limiting the discovery of improved long-term treatment strategies. To safely improve policy beyond clinician recommendations while ensuring that state-action trajectories remain in-distribution, we propose textitOffline Guarded Safe Reinforcement Learning (mathsfOGSRL), a theoretically grounded model-based offline RL framework. mathsfOGSRL introduces a novel dual constraint mechanism for improving policy with reliability and safety. First, the OOD guardian is established to specify clinically validated regions for safe policy exploration. By constraining optimization within these regions, it enables the reliable exploration of treatment strategies that outperform clinician behavior by leveraging the full patient state history, without drifting into unsupported state-action trajectories. Second, we introduce a safety cost constraint that encodes medical knowledge about physiological safety boundaries, providing domain-specific safeguards even in areas where training data might contain potentially unsafe interventions. Notably, we provide theoretical guarantees on safety and near-optimality: policies that satisfy these constraints remain in safe and reliable regions and achieve performance close to the best possible policy supported by the data.