mlsanityData to 7 Oct 2026

The hidden risks of temporal resampling in clinical reinforcement learning

arXiv 2602.06603 ↗First posted 6 Feb 2026cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data. To accommodate inherently irregular clinical records, researchers often resample the data into uniform time intervals before training (known as binning). However, discretised data presents the model with a fictional representation of clinical scenarios, especially where unpredictable decision timings are common. As these models lack robust trial evidence, we chose to explore the effects of this further by conducting an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. The simulator was modified to include stochastic intervals between decisions and used to generate a training dataset for offline RL. We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data, with 4-hour binning causing all agents to perform worse than the dataset's baseline. Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5-3x better returns than agents achieved in practice. We recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 124 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

40 of 124 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment25 Nov
  2. 2023End-to-end Offline Reinforcement Learning for Glycemia Control16 Oct
  3. 2022Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning5 Oct
  4. 2022Semi-Markov Offline Reinforcement Learning for Healthcare17 Mar
  5. 2021Offline reinforcement learning with uncertainty for treatment strategies in sepsis9 Jul
  6. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs31 May

Other references · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  10. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  12. 2019Batch Policy Learning under Constraintscited by 242
22 more
  1. 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 234
  2. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  3. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  4. 2016Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequencescited by 98
  5. 2021Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Seriescited by 96
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  7. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  8. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  9. 2020A Survey on Principles, Models and Methods for Learning from Irregularly Sampled Time Seriescited by 42
  10. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  11. 2019Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learningcited by 34
  12. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  13. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  14. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  15. 2018Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Datacited by 23
  16. 2021Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatmentcited by 16
  17. 2019Does the "Artificial Intelligence Clinician" learn optimal treatment strategies for sepsis in intensive care?cited by 9
  18. 2019Deep Reinforcement Learning for Optimal Critical Care Pain Management with Morphine using Dueling Double-Deep Q Networkscited by 8
  19. 2023Offline Learning of Closed-Loop Deep Brain Stimulation Controllers for Parkinson Disease Treatmentcited by 8
  20. 2020Identifying Distinct, Effective Treatments for Acute Hypotension with SODA-RL: Safely Optimized Diverse Accurate Reinforcement Learningcited by 4
  21. 2024Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcarecited by 4
  22. 2018Discretizing Logged Interaction Data Biases Learning for Decision-Makingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
  2. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetes
  3. 2026Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management
  4. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
  5. 2025Flexible Blood Glucose Control: Offline Reinforcement Learning from Human Feedback
  6. 2026GUIDE: Reinforcement Learning for Behavioral Action Support in Type 1 Diabetes
  7. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
  8. 2026From Static Risk to Dynamic Trajectories: Toward World-Model-Inspired Clinical Prediction