mlsanityData to 14 Oct 2026

A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processes

arXiv 2111.06784 ↗First posted 12 Nov 2021cs.LG stat.MLcited by 36

Not on the map. Why?

Abstract (arXiv)

We consider off-policy evaluation (OPE) in Partially Observable Markov Decision Processes (POMDPs), where the evaluation policy depends only on observable variables and the behavior policy depends on unobservable latent variables. Existing works either assume no unmeasured confounders, or focus on settings where both the observation and the state spaces are tabular. In this work, we first propose novel identification methods for OPE in POMDPs with latent confounders, by introducing bridge functions that link the target policy's value and the observed data distribution. We next propose minimax estimation methods for learning these bridge functions, and construct three estimators based on these estimated bridge functions, corresponding to a value function-based estimator, a marginalized importance sampling estimator, and a doubly-robust estimator. Our proposal permits general function approximation and is thus applicable to settings with continuous or large observation/state spaces. The nonasymptotic and asymptotic properties of the proposed estimators are investigated in detail.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 42 of its 77 references that are in the map. First posted on arXiv 12 Nov 2021.

Builds on

42 of 77 references.

References · 42

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2016OpenAI Gymcited by 1,736
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  8. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  9. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  10. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  11. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  12. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
30 more
  1. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  2. 2020Minimax Estimation of Conditional Moment Modelscited by 83
  3. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  4. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  5. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  6. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  7. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  8. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  9. 2020Semiparametric proximal causal inferencecited by 59
  10. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
  11. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  12. 2021Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restrictioncited by 55
  13. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 44
  14. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
  15. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  16. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  17. 2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 40
  18. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  19. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  20. 2020POPCORN: Partially Observed Prediction COnstrained ReiNforcement Learningcited by 36
  21. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  22. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  23. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  24. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  25. 2020Kernel Methods for Unobserved Confounding: Negative Controls, Proxies, and Instrumentscited by 28
  26. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  27. 2020The Variational Method of Momentscited by 23
  28. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  29. 2021Minimax Kernel Machine Learning for a Class of Doubly Robust Functionals with Application to Proximal Causal Inferencecited by 20
  30. 2021Proximal Causal Inference for Complex Longitudinal Studiescited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 6; 2024: 5; 2025: 11; 2026: 6.

36 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2023A Survey on Causal Reinforcement Learningcited by 45
  3. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  4. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  5. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  6. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  7. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 13
  8. 2022Spectral Representation Learning for Conditional Moment Modelscited by 12
  9. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  10. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  11. 2024On the Curses of Future and History in Future-dependent Value Functions for Off-policy Evaluationcited by 10
  12. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
24 more
  1. 2025Statistical Multicriteria Evaluation of LLM-Generated Textcited by 9
  2. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 8
  3. 2024Regularized DeepIV with Model Selectioncited by 7
  4. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
  5. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  6. 2024RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluationcited by 6
  7. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 6
  8. 2025Automatic Reward Shaping from Confounded Offline Datacited by 6
  9. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  10. 2023Bi-Level Offline Policy Optimization with Limited Explorationcited by 5
  11. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  12. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  13. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  14. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimatorscited by 3
  15. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  16. 2023Stackelberg Batch Policy Learningcited by 1
  17. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  18. 2023AI in Pharma for Personalized Sequential Decision-Making: Methods, Applications and Opportunitiescited by 0
  19. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0
  20. 2026Causal Imitation Learning Under Measurement Error and Distribution Shiftcited by 0
  21. 2026Proximal Mediation Analysis with Hidden Recanting Witnessescited by 0
  22. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0
  23. 2026Revealing After Overwriting: An Exponential POMDP OPE Lower Bound under History-Dependent Loggingcited by 0
  24. 2026What a Data Release Must Preserve for Off-Policy Evaluationcited by 0