mlsanityData to 7 Oct 2026

Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounders

arXiv 2007.13893 ↗First posted 27 Jul 2020cs.LG cs.AI stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation (OPE) in reinforcement learning is an important problem in settings where experimentation is limited, such as education and healthcare. But, in these very same settings, observed actions are often confounded by unobserved variables making OPE even more difficult. We study an OPE problem in an infinite-horizon, ergodic Markov decision process with unobserved confounders, where states and actions can act as proxies for the unobserved confounders. We show how, given only a latent variable model for states and actions, policy value can be identified from off-policy data. Our method involves two stages. In the first, we show how to use proxies to estimate stationary distribution ratios, extending recent work on breaking the curse of horizon to the confounded setting. In the second, we show optimal balancing can be combined with such learned ratios to obtain policy value while avoiding direct modeling of reward functions. We establish theoretical guarantees of consistency, and benchmark our method empirically.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 11; 2023: 8; 2024: 4; 2025: 5; 2026: 2.

40 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2021Universal Off-Policy Evaluationcited by 43
  3. 2023A Survey on Causal Reinforcement Learningcited by 43
  4. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  5. 2021Causal Reinforcement Learning using Observational and Interventional Datacited by 38
  6. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  7. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  8. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  9. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  10. 2021On Instrumental Variable Regression for Deep Offline Policy Evaluationcited by 22
  11. 2023Causal Reinforcement Learning: A Surveycited by 22
  12. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
28 more
  1. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  2. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  3. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  4. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  5. 2021Training a Resilient Q-Network against Observational Interferencecited by 9
  6. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  7. 2022Functional Generalized Empirical Likelihood Estimation for Conditional Moment Restrictionscited by 8
  8. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  9. 2024Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlapcited by 7
  10. 2021Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approachcited by 6
  11. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  12. 2021Causal Forecasting:Generalization Bounds for Autoregressive Modelscited by 5
  13. 2022A Reinforcement Learning Approach to Estimating Long-term Treatment Effectscited by 5
  14. 2023HOPE: Human-Centric Off-Policy Evaluation for E-Learning and Healthcarecited by 5
  15. 2023Estimation Beyond Data Reweighting: Kernel Method of Momentscited by 5
  16. 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
  17. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  18. 2023Offline Recommender System Evaluation under Unobserved Confoundingcited by 4
  19. 2024Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimizationcited by 4
  20. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  21. 2023Off-Policy Evaluation for Action-Dependent Non-Stationary Environmentscited by 2
  22. 2024Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneitycited by 1
  23. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applicationscited by 1
  24. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 1
  25. 2022Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approachcited by 0
  26. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  27. 2025The Sample Complexity of Online Strategic Decision Making with Information Asymmetry and Knowledge Transportabilitycited by 0
  28. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0