mlsanityData to 7 Oct 2026

Future-Dependent Value-Based Off-Policy Evaluation in POMDPs

arXiv 2207.13081 ↗First posted 26 Jul 2022cs.LG stat.MLcited by 20

Not on the map. Why?

Abstract (arXiv)

We study off-policy evaluation (OPE) for partially observable MDPs (POMDPs) with general function approximation. Existing methods such as sequential importance sampling estimators and fitted-Q evaluation suffer from the curse of horizon in POMDPs. To circumvent this problem, we develop a novel model-free OPE method by introducing future-dependent value functions that take future proxies as inputs. Future-dependent value functions play similar roles as classical value functions in fully-observable MDPs. We derive a new Bellman equation for future-dependent value functions as conditional moment equations that use history proxies as instrumental variables. We further propose a minimax learning method to learn future-dependent value functions using the new Bellman equation. We obtain the PAC result, which implies our OPE estimator is consistent as long as futures and histories contain sufficient information about latent states, and the Bellman completeness. Finally, we extend our methods to learning of dynamics and establish the connection between our approach and the well-known spectral learning methods in POMDPs.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 65 of its 123 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

65 of 123 references.

References · 65

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  7. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  8. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2019Batch Policy Learning under Constraintscited by 242
  12. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
38 more
  1. 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
  2. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  3. 2019Orthogonal Statistical Learningcited by 150
  4. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  5. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  6. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 138
  7. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  8. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  9. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  10. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  11. 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
  12. 2020Improper Learning for Non-Stochastic Controlcited by 83
  13. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  14. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  15. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  16. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 69
  17. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  18. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  19. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  20. 2018Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Predictioncited by 61
  21. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  22. 2020Semiparametric proximal causal inferencecited by 58
  23. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  24. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  25. 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 51
  26. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 43
  27. 2022PAC Reinforcement Learning for Predictive State Representationscited by 42
  28. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  29. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  30. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  31. 2016A PAC RL Algorithm for Episodic POMDPscited by 37
  32. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  33. 2018A confounding bridge approach for double negative control inference on causal effectscited by 36
  34. 2020POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learningcited by 36
  35. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  36. 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
  37. 2020Adversarial Estimation of Riesz Representerscited by 34
  38. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 3; 2025: 6; 2026: 3.

20 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  3. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  4. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  5. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  6. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 9
  7. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 7
  8. 2023SCOPE-RL: A Python Library for Offline Reinforcement Learning and Off-Policy Evaluationcited by 6
  9. 2024Regularized DeepIV with Model Selectioncited by 6
  10. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  11. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  12. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
8 more
  1. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  2. 2025To Distill or Decide? Understanding the Algorithmic Trade-off in Partially Observable Reinforcement Learningcited by 3
  3. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  4. 2026A Covering Framework for Offline POMDPs Learning using Belief Space Metriccited by 1
  5. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  6. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0
  7. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0
  8. 2026Exponential Hardness of Off-Policy Evaluation under History-Dependent Loggingcited by 0