mlsanityData to 7 Oct 2026

Semi-Markov Offline Reinforcement Learning for Healthcare

arXiv 2203.09365 ↗First posted 17 Mar 2022cs.LGcited by 18

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Reinforcement learning (RL) tasks are typically framed as Markov Decision Processes (MDPs), assuming that decisions are made at fixed time intervals. However, many applications of great importance, including healthcare, do not satisfy this assumption, yet they are commonly modelled as MDPs after an artificial reshaping of the data. In addition, most healthcare (and similar) problems are offline by nature, allowing for only retrospective studies. To address both challenges, we begin by discussing the Semi-MDP (SMDP) framework, which formally handles actions of variable timings. We next present a formal way to apply SMDP modifications to nearly any given value-based offline RL method. We use this theory to introduce three SMDP-based offline RL algorithms, namely, SDQN, SDDQN, and SBCQ. We then experimentally demonstrate that only these SMDP-based algorithms learn the optimal policy in variable-time environments, whereas their MDP counterparts do not. Finally, we apply our new algorithms to a real-world offline dataset pertaining to warfarin dosing for stroke prevention and demonstrate similar results.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 65 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

18 of 65 references.

References · 18

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  7. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  8. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  9. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  10. 2017Clinical Intervention Prediction and Understanding using Deep Networkscited by 35
  11. 2019Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learningcited by 34
  12. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
6 more
  1. 2020Clinician-in-the-Loop Decision Making: Reinforcement Learning with Near-Optimal Set-Valued Policiescited by 24
  2. 2017An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detectioncited by 16
  3. 2017On overfitting and asymptotic bias in batch reinforcement learning with partial observabilitycited by 16
  4. 2019Optimizing Sequential Medical Treatments with Auto-Encoding Heuristic Search in POMDPscited by 12
  5. 2018Steering Social Activity: A Stochastic Optimal Control Point Of Viewcited by 6
  6. 2020Reinforcement Learning in a Physics-Inspired Semi-Markov Environmentcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 7; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applicationscited by 17
  4. 2023Efficient Planning with Latent Diffusioncited by 11
  5. 2025Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RLcited by 5
  6. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3
  7. 2025Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learningcited by 3
  8. 2025MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representationcited by 2
  9. 2024Contrastive Diffuser: Planning Towards High Return States via Contrastive Learningcited by 1
  10. 2024Development and Validation of Heparin Dosing Policies Using an Offline Reinforcement Learning Algorithmcited by 1
  11. 2025Diffusion Policies with Offline and Inverse Reinforcement Learning for Promoting Physical Activity in Older Adults Using Wearable Sensorscited by 1
  12. 2025Treatment Stitching with Schr"odinger Bridge for Enhancing Offline Reinforcement Learning in Adaptive Treatment Strategiescited by 1
6 more
  1. 2026The hidden risks of temporal resampling in clinical reinforcement learningcited by 1
  2. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learningcited by 1
  3. 2024A Dynamical View of the Question of Whycited by 0
  4. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learningcited by 0
  5. 2025RAD: Retrieval High-quality Demonstrations to Enhance Decision-makingcited by 0
  6. 2026Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative Missingnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
  2. 2022Federated Offline Reinforcement Learning
  3. 2026Learning to Compress Time-to-Control: A Reinforcement Learning Framework for Chronic Disease Management
  4. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes
  5. 2025Test-Time Learning and Inference-Time Deliberation for Efficiency-First Offline Reinforcement Learning in Care Coordination and Population Health Management
  6. 2022Offline Policy Evaluation and Optimization under Confounding
  7. 2025Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data
  8. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints