# Evaluating covariate balance for long time horizon Markov decision processes

arXiv: https://arxiv.org/abs/2607.15080 · HTML: https://mlsanity.com/paper/2607.15080/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)
- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 11 of 11)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1911.06854/) (2019)
- [Benchmarks for Deep Off-Policy Evaluation](https://mlsanity.com/paper/2103.16596/) (2021)
- [A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units](https://mlsanity.com/paper/1704.06300/) (2017)
- [Model-Based Reinforcement Learning for Sepsis Treatment](https://mlsanity.com/paper/1811.09602/) (2018)
- [Towards Safe Mechanical Ventilation Treatment Using Deep Offline Reinforcement Learning](https://mlsanity.com/paper/2210.02552/) (2022)
- [Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning](https://mlsanity.com/paper/2405.14335/) (2024)
- [Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters](https://mlsanity.com/paper/1807.01066/) (2018)
- [A Conservative Q-Learning approach for handling distribution shift in sepsis treatment strategies](https://mlsanity.com/paper/2203.13884/) (2022)

## Abstract (arXiv)

This article explores the application of covariate balance diagnostics for detecting the presence of hidden confounding/model miss-specification in studies applying offline reinforcement learning (RL) to deriving optimal treatment recommendations. The results demonstrate that, either there is a high risk of bias within existing offline RL studies for treatment recommendations and/or, existing covariate balance metrics are not sufficient to assess such studies. Regardless, existing offline RL studies cannot be concluded as being statistically robust. The conclusions propose future research directions for obtaining more methodologically robust applications of offline RL to treatment recommendation problems.