mlsanityData to 14 Oct 2026

Off-Policy Evaluation Under Nonignorable Missing Data

arXiv 2507.06961 ↗First posted 9 Jul 2025stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Off-Policy Evaluation (OPE) aims to estimate the value of a target policy using offline data collected from potentially different policies. In real-world applications, however, logged data often suffers from missingness. While OPE has been extensively studied in the literature, a theoretical understanding of how missing data affects OPE results remains unclear. In this paper, we investigate OPE in the presence of monotone missingness and theoretically demonstrate that the value estimates remain unbiased under ignorable missingness but can be biased under nonignorable (informative) missingness. To retain the consistency of value estimation, we propose an inverse probability weighted value estimator and conduct statistical inference to quantify the uncertainty of the estimates. Through a series of numerical experiments, we empirically demonstrate that our proposed estimator yields a more reliable value inference under missing data.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 30 of its 78 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

30 of 78 references.

References · 30

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  8. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  9. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  10. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  11. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  12. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
18 more
  1. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  2. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  3. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  4. 2018Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertisingcited by 65
  5. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  6. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  7. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  8. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  9. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  10. 2019Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learningcited by 34
  11. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  12. 2018Model-Based Reinforcement Learning for Sepsis Treatmentcited by 21
  13. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  14. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  15. 2019Robust Reinforcement Learning in POMDPs with Incomplete and Noisy Observationscited by 10
  16. 2016Semiparametric Estimation with Data Missing Not at Random Using an Instrumental Variablecited by 9
  17. 2020Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluationcited by 8
  18. 2019A Versatile Estimation Procedure without Estimating the Nonignorable Missingness Mechanismcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Off-Policy Evaluation and Learning for Survival Outcomes under Censoringcited by 0
  2. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0