mlsanityData to 14 Oct 2026

Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Random

arXiv 2606.20206 ↗First posted 18 Jun 2026stat.ML cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

In offline Reinforcement Learning, immediate rewards in logged batch data are often unobserved due to sparse or irregular record-keeping, or censored beyond certain reward values. This issue arises in practical settings, including health care and marketing. We investigate off-policy evaluation (OPE) in finite-horizon Markov decision processes when rewards are missing not at random (MNAR), which breaks ignorability and induces selection bias even after conditioning on states and actions. To address this, we formalize a reward-dependent propensity model and use future states as shadow variables to identify the full-data conditional mean reward. We further introduce a bridge function that recovers the conditional mean reward without explicitly modeling the MNAR mechanism, and estimate it via a min-max procedure to avoid double sampling. Building upon these identification results, we propose an Fitted-Q-Evaluation-style estimator that propagates the recovered rewards while allowing target policies to depend on past missingness indicators. Finally, we establish consistency and finite-sample error bounds for our OPE estimator, and show through experiments the strong performance of our method compared to existing methods on simulated and MIMIC-III Sepsis data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 80 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

44 of 80 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories12 Oct
  2. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  4. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes28 Oct
  5. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 36

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  5. 2019Batch Policy Learning under Constraintscited by 244
  6. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  7. 2019Orthogonal Statistical Learningcited by 151
  8. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  9. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  10. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  11. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  12. 2017Sobolev Norm Learning Rates for Regularized Least-Squares Algorithmcited by 114
24 more
  1. 2020An Introduction to Proximal Causal Learningcited by 95
  2. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  3. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  4. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  5. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  6. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  7. 2020Semiparametric proximal causal inferencecited by 59
  8. 2018Graphical Models for Processing Missing Datacited by 58
  9. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
  10. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  11. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  12. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  13. 2016Generalizing Skills with Semi-Supervised Reinforcement Learningcited by 18
  14. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  15. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  16. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  17. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 10
  18. 2016Semiparametric Estimation with Data Missing Not at Random Using an Instrumental Variablecited by 9
  19. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 8
  20. 2023A Policy Gradient Method for Confounded POMDPscited by 6
  21. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  22. 2025Off-Policy Evaluation Under Nonignorable Missing Datacited by 2
  23. 2015Off-Policy Reward Shaping with Ensemblescited by 1
  24. 2025Evaluating and Learning Optimal Dynamic Treatment Regimes under Truncation by Deathcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  2. 2024Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy
  3. 2024Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation
  4. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap
  5. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
  6. 2026Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
  7. 2022Off-Policy Risk Assessment in Markov Decision Processes
  8. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation