# On the Challenges of using Reinforcement Learning in Precision Drug Dosing: Delay and Prolongedness of Action Effects

arXiv: https://arxiv.org/abs/2301.00512 · HTML: https://mlsanity.com/paper/2301.00512/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 2 Jan 2023.

## Builds on (most cited first, 8 of 8)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [On Improving Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1804.06309/) (2018)
- [Learning to Perform Physics Experiments via Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01843/) (2016)
- [Deep Reinforcement Learning for Closed-Loop Blood Glucose Control](https://mlsanity.com/paper/2009.09051/) (2020)
- [Representation and Reinforcement Learning for Personalized Glycemic Control in Septic Patients](https://mlsanity.com/paper/1712.00654/) (2017)
- [Deep Reinforcement Learning for Optimal Critical Care Pain Management with Morphine using Dueling Double-Deep Q Networks](https://mlsanity.com/paper/1904.11115/) (2019)
- [Optimizing Warfarin Dosing using Deep Reinforcement Learning](https://mlsanity.com/paper/2202.03486/) (2022)

## Built on by (most cited first, 4 of 4)

- [From Silos to Systems: Process-Oriented Hazard Analysis for AI Systems](https://mlsanity.com/paper/2410.22526/) (2024)
- [Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AI](https://mlsanity.com/paper/2508.21101/) (2025)
- [An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning](https://mlsanity.com/paper/2403.07566/) (2024)
- [Wavelet Predictive Representations for Non-Stationary Reinforcement Learning](https://mlsanity.com/paper/2510.04507/) (2025)

## Abstract (arXiv)

Drug dosing is an important application of AI, which can be formulated as a Reinforcement Learning (RL) problem. In this paper, we identify two major challenges of using RL for drug dosing: delayed and prolonged effects of administering medications, which break the Markov assumption of the RL framework. We focus on prolongedness and define PAE-POMDP (Prolonged Action Effect-Partially Observable Markov Decision Process), a subclass of POMDPs in which the Markov assumption does not hold specifically due to prolonged effects of actions. Motivated by the pharmacology literature, we propose a simple and effective approach to converting drug dosing PAE-POMDPs into MDPs, enabling the use of the existing RL algorithms to solve such problems. We validate the proposed approach on a toy task, and a challenging glucose control task, for which we devise a clinically-inspired reward function. Our results demonstrate that: (1) the proposed method to restore the Markov assumption leads to significant improvements over a vanilla baseline; (2) the approach is competitive with recurrent policies which may inherently capture the prolonged effect of actions; (3) it is remarkably more time and memory efficient than the recurrent baseline and hence more suitable for real-time dosing control systems; and (4) it exhibits favorable qualitative behavior in our policy analysis.