mlsanityData to 14 Oct 2026

Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability

arXiv 2110.12343 ↗First posted 24 Oct 2021cs.LG math.ST stat.MEcited by 24

Not on the map. Why?

Abstract (arXiv)

We consider off-policy evaluation of dynamic treatment rules under sequential ignorability, given an assumption that the underlying system can be modeled as a partially observed Markov decision process (POMDP). We propose an estimator, partial history importance weighting, and show that it can consistently estimate the stationary mean rewards of a target policy given long enough draws from the behavior policy. We provide an upper bound on its error that decays polynomially in the number of observations (i.e., the number of trajectories times their length), with an exponent that depends on the overlap of the target and behavior policies, and on the mixing time of the underlying system. Furthermore, we show that this rate of convergence is minimax given only our assumptions on mixing and overlap. Our results establish that off-policy evaluation in POMDPs is strictly harder than off-policy evaluation in (fully observed) Markov decision processes, but strictly easier than model-free off-policy evaluation.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 16 of its 90 references that are in the map. First posted on arXiv 24 Oct 2021.

Builds on

16 of 90 references.

References · 16

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  4. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  5. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  6. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  7. 2016Optimal and Adaptive Off-policy Evaluation in Contextual Banditscited by 111
  8. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  9. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  10. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  11. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  12. 2019Learning When-to-Treat Policiescited by 49
4 more
  1. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  2. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  3. 2015Counterfactual Risk Minimization: Learning from Logged Bandit Feedbackcited by 33
  4. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 2; 2024: 5; 2025: 6; 2026: 2.

24 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2023A Survey on Causal Reinforcement Learningcited by 45
  3. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  4. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  5. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  6. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  7. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  8. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 10
  9. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  10. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
  11. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 8
  12. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 8
12 more
  1. 2021Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approachcited by 6
  2. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  3. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  4. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  5. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 4
  6. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  7. 2025A Graphical Approach to State Variable Selection in Off-policy Learningcited by 1
  8. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  9. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  10. 2026A Covering Framework for Offline POMDPs Learning using Belief Space Metriccited by 1
  11. 2023Asymptotic Inference for Multi-Stage Stationary Treatment Policy with Variable Selectioncited by 0
  12. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0