mlsanityData to 14 Oct 2026

Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry

arXiv 2608.30442 ↗First posted 31 Aug 2026cs.LG stat.MLcited by 0

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Recent offline reinforcement learning (RL) studies report policies that outperform physician decisions on clinical outcomes. We conduct a systematic, partially crossed evaluation of five offline RL algorithm families and 14 reward designs in 44,894 post-2018 acute ischemic stroke patients from a nationwide registry (N = 129,033). Standard Fitted Q-Evaluation (FQE) yields an apparent policy-improvement estimate of +0.0069; adding an Early Neurological Deterioration penalty increases it to +0.0101. We identify reward-embedded confounding, in which a proxy terminal reward encodes baseline severity and prognosis as well as treatment efficacy. A 2 x 2 factorial analysis finds that terminal reward confounding accounts for 218.6% of the observed signal change, so its removal overshoots the null. After DML-inspired GBM reward residualization, the FQE estimate attenuates to +0.0033 (p = 0.132), and full deconfounding yields +0.0025 (p = 0.291). FQE-based diagnostics, T-learner analyses, and direct recurrence analyses converge away from a clinically meaningful aggregate improvement. A 1-year mRS factorial analysis replicates the attenuation. We provide an empirically motivated six-step evaluation checklist. NIHSS-stratified heterogeneity is hypothesis-generating for prospective trial design; hospital-level disagreement does not persist after full reward deconfounding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 36 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

14 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination28 May
  2. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications15 Feb
  3. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings23 Jul

Other references · 11

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 600
  6. 2017Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learningcited by 478
  7. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  8. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  9. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  10. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
  11. 2018Deconfounding Reinforcement Learning in Observational Settingscited by 36

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functions
  2. 2022Offline Policy Evaluation and Optimization under Confounding
  3. 2024OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment
  4. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  5. 2026The Evidence Ladder for Reinforcement Learning in Healthcare: From Retrospective Policies to Trusted Interventions
  6. 2022Semi-Markov Offline Reinforcement Learning for Healthcare
  7. 2025Reinforcement Learning enhanced Online Adaptive Clinical Decision Support via Digital Twin powered Policy and Treatment Effect optimized Reward
  8. 2026Learning Clinical-Trial Strategy: Offline Policy Training for Decision Agents