mlsanityData to 7 Oct 2026

Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Models

arXiv 2209.10064 ↗First posted 21 Sep 2022stat.ML cs.LG math.STcited by 18

Not on the map. Why?

Abstract (arXiv)

We study the problem of off-policy evaluation (OPE) for episodic Partially Observable Markov Decision Processes (POMDPs) with continuous states. Motivated by the recently proposed proximal causal inference framework, we develop a non-parametric identification result for estimating the policy value via a sequence of so-called V-bridge functions with the help of time-dependent proxy variables. We then develop a fitted-Q-evaluation-type algorithm to estimate V-bridge functions recursively, where a non-parametric instrumental variable (NPIV) problem is solved at each step. By analyzing this challenging sequential NPIV problem, we establish the finite-sample error bounds for estimating the V-bridge functions and accordingly that for evaluating the policy value, in terms of the sample size, length of horizon and so-called (local) measure of ill-posedness at each step. To the best of our knowledge, this is the first finite-sample error bound for OPE in POMDPs under non-parametric models.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 19 of its 66 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

19 of 66 references.

References · 19

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2019Orthogonal Statistical Learningcited by 150
  3. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 138
  4. 2020An Introduction to Proximal Causal Learningcited by 91
  5. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  6. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  7. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  8. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  9. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  10. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  11. 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
  12. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
7 more
  1. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  2. 2015Optimal Sup-norm Rates and Uniform Inference on Nonlinear Functionals of Nonparametric IV Regressioncited by 30
  3. 2017Early stopping for kernel boosting algorithms: A general analysis with localized complexitiescited by 27
  4. 2020Kernel Methods for Unobserved Confounding: Negative Controls, Proxies, and Instrumentscited by 27
  5. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
  6. 2021Proximal Causal Inference for Complex Longitudinal Studiescited by 16
  7. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 4; 2024: 1; 2025: 3; 2026: 5.

18 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  3. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  4. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  5. 2023Source Condition Double Robust Inference on Functionals of Inverse Problemscited by 8
  6. 2023A Policy Gradient Method for Confounded POMDPscited by 6
  7. 2025Automatic Reward Shaping from Confounded Offline Datacited by 6
  8. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  9. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  10. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  11. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  12. 2026Confounding Robust Continuous Control via Automatic Reward Shapingcited by 2
6 more
  1. 2023Personalized Pricing with Invalid Instrumental Variables: Identification, Estimation, and Policy Learningcited by 1
  2. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 1
  3. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  4. 2026Causal Imitation Learning Under Measurement Error and Distribution Shiftcited by 0
  5. 2026CoSToM:Causal-oriented Steering for Intrinsic Theory-of-Mind Alignment in Large Language Modelscited by 0
  6. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0