# Towards Real-World Applications of Personalized Anesthesia Using Policy Constraint Q Learning for Propofol Infusion Control

arXiv: https://arxiv.org/abs/2303.10180 · HTML: https://mlsanity.com/paper/2303.10180/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 17 Mar 2023.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Mildly Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2206.04745/) (2022)
- [d3rlpy: An Offline Deep Reinforcement Learning Library](https://mlsanity.com/paper/2111.03788/) (2021)
- [Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1705.08422/) (2017)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [Off-Policy Evaluation via Off-Policy Classification](https://mlsanity.com/paper/1906.01624/) (2019)
- [Federated Learning for Privacy Preservation in Smart Healthcare Systems: A Comprehensive Survey](https://mlsanity.com/paper/2203.09702/) (2022)
- [Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes](https://mlsanity.com/paper/2204.03376/) (2022)
- [Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation](https://mlsanity.com/paper/2005.09059/) (2020)
- [Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning](https://mlsanity.com/paper/2210.02552/) (2022)
- [Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space](https://mlsanity.com/paper/2206.11190/) (2022)
- [Multi-Task Time Series Analysis applied to Drug Response Modelling](https://mlsanity.com/paper/1903.08970/) (2019)

## Built on by (most cited first, 2 of 2)

- [Treatment Stitching with Schr"odinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategies](https://mlsanity.com/paper/2511.12075/) (2025)
- [Near-Constant Strong Violation and Last-Iterate Convergence for Online CMDPs via Decaying Safety Margins](https://mlsanity.com/paper/2602.10917/) (2026)

## Abstract (arXiv)

Automated anesthesia promises to enable more precise and personalized anesthetic administration and free anesthesiologists from repetitive tasks, allowing them to focus on the most critical aspects of a patient's surgical care. Current research has typically focused on creating simulated environments from which agents can learn. These approaches have demonstrated good experimental results, but are still far from clinical application. In this paper, Policy Constraint Q-Learning (PCQL), a data-driven reinforcement learning algorithm for solving the problem of learning anesthesia strategies on real clinical datasets, is proposed. Conservative Q-Learning was first introduced to alleviate the problem of Q function overestimation in an offline context. A policy constraint term is added to agent training to keep the policy distribution of the agent and the anesthesiologist consistent to ensure safer decisions made by the agent in anesthesia scenarios. The effectiveness of PCQL was validated by extensive experiments on a real clinical anesthesia dataset. Experimental results show that PCQL is predicted to achieve higher gains than the baseline approach while maintaining good agreement with the reference dose given by the anesthesiologist, using less total dose, and being more responsive to the patient's vital signs. In addition, the confidence intervals of the agent were investigated, which were able to cover most of the clinical decisions of the anesthesiologist. Finally, an interpretable method, SHAP, was used to analyze the contributing components of the model predictions to increase the transparency of the model.