mlsanityData to 14 Oct 2026

An Instrumental Variable Approach to Confounded Off-Policy Evaluation

arXiv 2212.14468 ↗First posted 29 Dec 2022stat.ML cs.LG stat.MEcited by 20

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation (OPE) is a method for estimating the return of a target policy using some pre-collected observational data generated by a potentially different behavior policy. In some cases, there may be unmeasured variables that can confound the action-reward or action-next-state relationships, rendering many existing OPE approaches ineffective. This paper develops an instrumental variable (IV)-based method for consistent OPE in confounded Markov decision processes (MDPs). Similar to single-stage decision making, we show that IV enables us to correctly identify the target policy's value in infinite horizon settings as well. Furthermore, we propose an efficient and robust value estimator and illustrate its effectiveness through extensive simulations and analysis of real data from a world-leading short-video platform.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 56 of its 89 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

56 of 89 references.

References · 56

  1. 2017Causal Effect Inference with Deep Latent-Variable Modelscited by 410
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  3. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  4. 2019Batch Policy Learning under Constraintscited by 244
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  6. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  9. 2016Identifying Causal Effects With Proxy Variables of an Unmeasured Confoundercited by 139
  10. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  11. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  12. 2020An Introduction to Proximal Causal Learningcited by 95
38 more
  1. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  2. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  3. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  4. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  5. 2020Sample-Efficient Reinforcement Learning of Undercomplete POMDPscited by 66
  6. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  7. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  8. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  9. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  10. 2020Semiparametric proximal causal inferencecited by 59
  11. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
  12. 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 52
  13. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  14. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  15. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 44
  16. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  17. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
  18. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 41
  19. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  20. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  21. 2021Deep Proxy Causal Learning and its Application to Confounded Bandit Policy Evaluationcited by 40
  22. 2016A PAC RL Algorithm for Episodic POMDPscited by 37
  23. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  24. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  25. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  26. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  27. 2018Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methodscited by 33
  28. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  29. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  30. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  31. 2022Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Modelcited by 30
  32. 2016Contextual Bandits with Latent Confounders: An NMF Approachcited by 26
  33. 2019Importance Resampling for Off-policy Predictioncited by 26
  34. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  35. 2016Bounded, efficient and multiply robust estimation of average treatment effects using instrumental variablescited by 23
  36. 2019A semiparametric instrumental variable approach to optimal treatment regimes under endogeneitycited by 22
  37. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  38. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 9; 2026: 3.

20 papers.

Sort:

By area:

  1. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  2. 2025Statistical Tractability of Off-policy Evaluation of History-dependent Policies in POMDPscited by 8
  3. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  4. 2024RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluationcited by 6
  5. 2024Learning Decision Policies with Instrumental Variables through Double Machine Learningcited by 5
  6. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  7. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  8. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  9. 2023Offline Recommender System Evaluation under Unobserved Confoundingcited by 4
  10. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  11. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  12. 2025Off-Policy Evaluation Under Nonignorable Missing Datacited by 2
8 more
  1. 2025A Graphical Approach to State Variable Selection in Off-policy Learningcited by 1
  2. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  3. 2023A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learningcited by 0
  4. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  5. 2025Confounding is a Pervasive Problem in Real World Recommender Systemscited by 0
  6. 2025Differentially Private Two-Stage Gradient Descent for Instrumental Variable Regressioncited by 0
  7. 2026Robust Sequential Experimental Design for A/B Testingcited by 0
  8. 2026Off-Policy Evaluation for Missingness-Aware Policies in MDPs with Rewards Missing Not at Randomcited by 0