# Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes

arXiv: https://arxiv.org/abs/2204.03376 · HTML: https://mlsanity.com/paper/2204.03376/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 16 of its 74 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Placed in

- Offline RL and imitation learning from data / [Safe offline RL](https://mlsanity.com/offlinerl/safe-offline-rl/)

## Builds on (most cited first, 16 of 16)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Benchmarks for Deep Off-Policy Evaluation](https://mlsanity.com/paper/2103.16596/) (2021)
- [Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings](https://mlsanity.com/paper/2107.11003/) (2021)
- [Deep Reinforcement Learning for Closed-Loop Blood Glucose Control](https://mlsanity.com/paper/2009.09051/) (2020)
- [Basal Glucose Control in Type 1 Diabetes using Deep Reinforcement Learning: An In Silico Validation](https://mlsanity.com/paper/2005.09059/) (2020)
- [Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons](https://mlsanity.com/paper/2202.13163/) (2022)
- [Offline reinforcement learning with uncertainty for treatment strategies in sepsis](https://mlsanity.com/paper/2107.04491/) (2021)
- [A Dual-Hormone Closed-Loop Delivery System for Type 1 Diabetes Using Deep Reinforcement Learning](https://mlsanity.com/paper/1910.04059/) (2019)

## Built on by (most cited first, 23 of 23)

- [TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents](https://mlsanity.com/paper/2404.12530/) (2024)
- [MetaboNet: The Largest Publicly Available Consolidated Dataset for Type 1 Diabetes Management](https://mlsanity.com/paper/2601.11505/) (2026)
- [Generating Personalized Insulin Treatments Strategies with Deep Conditional Generative Time Series Models](https://mlsanity.com/paper/2309.16521/) (2023)
- [Mitigating Deep Reinforcement Learning Backdoors in the Neural Activation Space](https://mlsanity.com/paper/2407.15168/) (2024)
- [Offline Reinforcement Learning with Behavioral Supervisor Tuning](https://mlsanity.com/paper/2404.16399/) (2024)
- [ADESSE: Advice Explanations in Complex Repeated Decision-Making Environments](https://mlsanity.com/paper/2405.20705/) (2024)
- [Towards Real-World Applications of Personalized Anesthesia Using Policy Constraint Q Learning for Propofol Infusion Control](https://mlsanity.com/paper/2303.10180/) (2023)
- [End-to-end Offline Reinforcement Learning for Glycemia Control](https://mlsanity.com/paper/2310.10312/) (2023)
- [Basal-Bolus Advisor for Type 1 Diabetes (T1D) Patients Using Multi-Agent Reinforcement Learning (RL) Methodology](https://mlsanity.com/paper/2307.08897/) (2023)
- [Methodology for Interpretable Reinforcement Learning for Optimizing Mechanical Ventilation](https://mlsanity.com/paper/2404.03105/) (2024)
- [Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations](https://mlsanity.com/paper/2412.18781/) (2024)
- [The Role of Artificial Intelligence in Enhancing Insulin Recommendations and Therapy Outcomes](https://mlsanity.com/paper/2503.18592/) (2025)
- [ExOSITO: Explainable Off-Policy Learning with Side Information for Intensive Care Unit Blood Test Orders](https://mlsanity.com/paper/2504.17277/) (2025)
- [PrivORL: Differentially Private Synthetic Dataset for Offline Reinforcement Learning](https://mlsanity.com/paper/2512.07342/) (2025)
- [An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learning](https://mlsanity.com/paper/2403.07566/) (2024)
- [Blood Glucose Control Via Pre-trained Counterfactual Invertible Neural Networks](https://mlsanity.com/paper/2405.17458/) (2024)
- [Dual Alignment Maximin Optimization for Offline Model-based RL](https://mlsanity.com/paper/2502.00850/) (2025)
- [Personalised Insulin Adjustment with Reinforcement Learning: An In-Silico Validation for People with Diabetes on Intensive Insulin Treatment](https://mlsanity.com/paper/2505.14477/) (2025)
- [Towards an Adaptive Social Game-Playing Robot: An Offline Reinforcement Learning-Based Framework](https://mlsanity.com/paper/2509.16858/) (2025)
- [Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control](https://mlsanity.com/paper/2510.13358/) (2025)
- [Diabetes Lifestyle Medicine Treatment Assistance Using Reinforcement Learning](https://mlsanity.com/paper/2510.26807/) (2025)
- [GUIDE: Reinforcement Learning for Behavioral Action Support in Type 1 Diabetes](https://mlsanity.com/paper/2604.00385/) (2026)
- [PhiBE-Q-Learning: Bridging Off-Policy Reinforcement Learning and Continuous-Time Control](https://mlsanity.com/paper/2606.21925/) (2026)

## Abstract (arXiv)

The widespread adoption of effective hybrid closed loop systems would represent an important milestone of care for people living with type 1 diabetes (T1D). These devices typically utilise simple control algorithms to select the optimal insulin dose for maintaining blood glucose levels within a healthy range. Online reinforcement learning (RL) has been utilised as a method for further enhancing glucose control in these devices. Previous approaches have been shown to reduce patient risk and improve time spent in the target range when compared to classical control algorithms, but are prone to instability in the learning process, often resulting in the selection of unsafe actions. This work presents an evaluation of offline RL for developing effective dosing policies without the need for potentially dangerous patient interaction during training. This paper examines the utility of BCQ, CQL and TD3-BC in managing the blood glucose of the 30 virtual patients available within the FDA-approved UVA/Padova glucose dynamics simulator. When trained on less than a tenth of the total training samples required by online RL to achieve stable performance, this work shows that offline RL can significantly increase time in the healthy blood glucose range from 61.6 +- 0.3% to 65.3 +/- 0.5% when compared to the strongest state-of-art baseline (p < 0.001). This is achieved without any associated increase in low blood glucose events. Offline RL is also shown to be able to correct for common and challenging control scenarios such as incorrect bolus dosing, irregular meal timings and compression errors.