mlsanityData to 7 Oct 2026

Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning

arXiv 2002.04518 ↗First posted 11 Feb 2020cs.LG math.OC stat.MLcited by 56

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Off-policy evaluation of sequential decision policies from observational data is necessary in applications of batch reinforcement learning such as education and healthcare. In such settings, however, unobserved variables confound observed actions, rendering exact evaluation of new policies impossible, i.e., unidentifiable. We develop a robust approach that estimates sharp bounds on the (unidentifiable) value of a given policy in an infinite-horizon problem given data from another policy with unobserved confounding, subject to a sensitivity model. We consider stationary or baseline unobserved confounding and compute bounds by optimizing over the set of all stationary state-occupancy ratios that agree with a new partially identified estimating equation and the sensitivity model. We prove convergence to the sharp bounds as we collect more confounded data. Although checking set membership is a linear program, the support function is given by a difficult nonconvex optimization problem. We develop approximations based on nonconvex projected gradient descent and demonstrate the resulting bounds empirically.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 11 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 19; 2023: 5; 2024: 7; 2025: 9; 2026: 6.

56 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2021Partial Counterfactual Identification from Observational and Experimental Datacited by 64
  3. 2021Universal Off-Policy Evaluationcited by 43
  4. 2023A Survey on Causal Reinforcement Learningcited by 43
  5. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  6. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningcited by 35
  7. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  8. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  9. 2021Doubly-Valid/Doubly-Sharp Sensitivity Analysis for Causal Inference with Unmeasured Confoundingcited by 28
  10. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  11. 2022Stochastic Causal Programming for Bounding Treatment Effectscited by 25
  12. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
38 more
  1. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  2. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  3. 2021Invariant Policy Learning: A Causal Perspectivecited by 17
  4. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
  5. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  6. 2023Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Modelcited by 14
  7. 2021Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variablecited by 13
  8. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  9. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  10. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  11. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  12. 2022Instrumental Variables in Causal Inference and Machine Learning: A Surveycited by 10
  13. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  14. 2022Strategic Decision-Making in the Presence of Information Asymmetry: Provably Efficient RL with Algorithmic Instrumentscited by 9
  15. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  16. 2021Ambiguous Dynamic Treatment Regimes: A Reinforcement Learning Approachcited by 6
  17. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  18. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  19. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  20. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  21. 2022Generalizing Off-Policy Evaluation From a Causal Perspective For Sequential Decision-Makingcited by 4
  22. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  23. 2022Causal Deep Reinforcement Learning Using Observational Datacited by 3
  24. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  25. 2024OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimatorscited by 3
  26. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  27. 2024Predictive Performance Comparison of Decision Policies Under Confoundingcited by 2
  28. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  29. 2022Data-Driven Influence Functions for Optimization-Based Causal Inferencecited by 1
  30. 2023Personalized Pricing with Invalid Instrumental Variables: Identification, Estimation, and Policy Learningcited by 1
  31. 2025Off-Policy Evaluation for Sequential Persuasion Process with Unobserved Confoundingcited by 1
  32. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  33. 2025Model-Based Reinforcement Learning Under Confoundingcited by 1
  34. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applicationscited by 1
  35. 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learningcited by 1
  36. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  37. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  38. 2024Bounding Causal Effects with Leaky Instrumentscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Reliable Off-policy Evaluation for Reinforcement Learning
  2. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes
  3. 2020Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learning
  4. 2020Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
  5. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning
  6. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  7. 2021Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithm
  8. 2020Provably Good Batch Reinforcement Learning Without Great Exploration