mlsanityData to 7 Oct 2026

Exploring Time-Step Size in Reinforcement Learning for Sepsis Treatment

arXiv 2511.20913 ↗First posted 25 Nov 2025cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Existing studies on reinforcement learning (RL) for sepsis management have mostly followed an established problem setup, in which patient data are aggregated into 4-hour time steps. Although concerns have been raised regarding the coarseness of this time-step size, which might distort patient dynamics and lead to suboptimal treatment policies, the extent to which this is a problem in practice remains unexplored. In this work, we conducted empirical experiments for a controlled comparison of four time-step sizes (Delta t!=!1,2,4,8 h) on this domain, following an identical offline RL pipeline. To enable a fair comparison across time-step sizes, we designed action re-mapping methods that allow for evaluation of policies on datasets with different time-step sizes, and conducted cross-Delta t model selections under two policy learning setups. Our goal was to quantify how time-step size influences state representation learning, behavior cloning, policy training, and off-policy evaluation. Our results show that performance trends across Delta t vary as learning setups change, while policies learned at finer time-step sizes (Delta t = 1 h and 2 h) using a static behavior policy achieve the overall best performance and stability. Our work highlights time-step size as a core design choice in offline RL for healthcare and provides evidence supporting alternatives beyond the conventional 4-hour setup.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 33 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

14 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings23 Jul
  2. 2021Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs31 May

Other references · 12

  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2020Approximate information state for approximate planning and reinforcement learning in partially observed systemscited by 67
  4. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 41
  5. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  6. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  7. 2018Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matterscited by 13
  8. 2019Does the "Artificial Intelligence Clinician" learn optimal treatment strategies for sepsis in intensive care?cited by 9
  9. 2024ICU-Sepsis: A Benchmark MDP Built from Real Medical Datacited by 6
  10. 2020Trajectory Inspection: A Method for Iterative Clinician-Driven Design of Reinforcement Learning Studiescited by 5
  11. 2018Discretizing Logged Interaction Data Biases Learning for Decision-Makingcited by 3
  12. 2024Identifying Differential Patient Care Through Inverse Intent Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Formally Verifying and Explaining Sepsis Treatment Policies with COOL-MCcited by 3
  2. 2026The hidden risks of temporal resampling in clinical reinforcement learningcited by 1
  3. 2026Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingnesscited by 0
  4. 2026Insulin4RL: Real-Time Insulin Management in the Intensive Care Unit for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline reinforcement learning with uncertainty for treatment strategies in sepsis
  2. 2022Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
  3. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
  4. 2025Exploring a Graph-based Approach to Offline Reinforcement Learning for Sepsis Treatment
  5. 2026Offline Reinforcement Learning for Hemodynamic Management of Sepsis in the ICU: a MIMIC-IV Study with Dual Off-Policy Evaluation
  6. 2024Empowering Clinicians with Medical Decision Transformers: A Framework for Sepsis Treatment
  7. 2026Learning Preference-Based Objectives from Clinical Narratives for Dynamic Sepsis Treatment
  8. 2022A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies