mlsanityData to 14 Oct 2026

Evaluating covariate balance for long time horizon Markov decision processes

arXiv 2607.15080 ↗First posted 16 Jul 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

This article explores the application of covariate balance diagnostics for detecting the presence of hidden confounding/model miss-specification in studies applying offline reinforcement learning (RL) to deriving optimal treatment recommendations. The results demonstrate that, either there is a high risk of bias within existing offline RL studies for treatment recommendations and/or, existing covariate balance metrics are not sufficient to assess such studies. Regardless, existing offline RL studies cannot be concluded as being statistically robust. The conclusions propose future research directions for obtaining more methodologically robust applications of offline RL to treatment recommendation problems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 11 of its 29 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

11 of 29 references.

References · 11

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  3. 2019Batch Policy Learning under Constraintscited by 244
  4. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  5. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  6. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  7. 2018Model-Based Reinforcement Learning for Sepsis Treatmentcited by 21
  8. 2022Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learningcited by 16
  9. 2024Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learningcited by 16
  10. 2018Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matterscited by 13
  11. 2022A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategiescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Projected State-action Balancing Weights for Offline Reinforcement Learning
  2. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  3. 2022Offline Policy Evaluation and Optimization under Confounding
  4. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
  5. 2021False Correlation Reduction for Offline Reinforcement Learning
  6. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  7. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  8. 2026Less is More: Clustered Cross-Covariance Control for Offline RL