mlsanityData to 14 Oct 2026

Offline Policy Evaluation and Optimization under Confounding

arXiv 2211.16583 ↗First posted 29 Nov 2022stat.ML cs.LGcited by 12

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Evaluating and optimizing policies in the presence of unobserved confounders is a problem of growing interest in offline reinforcement learning. Using conventional methods for offline RL in the presence of confounding can not only lead to poor decisions and poor policies, but also have disastrous effects in critical applications such as healthcare and education. We map out the landscape of offline policy evaluation for confounded MDPs, distinguishing assumptions on confounding based on whether they are memoryless and on their effect on the data-collection policies. We characterize settings where consistent value estimates are provably not achievable, and provide algorithms with guarantees to instead estimate lower bounds on the value. When consistent estimates are achievable, we provide algorithms for value estimation with sample complexity guarantees. We also present new algorithms for offline policy improvement and prove local convergence guarantees. Finally, we experimentally evaluate our algorithms on both a gridworld environment and a simulated healthcare setting of managing sepsis patients. In gridworld, our model-based method provides tighter lower bounds than existing methods, while in the sepsis simulator, our methods significantly outperform confounder-oblivious benchmarks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 51 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

24 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes18 Sep
  2. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes26 May
  3. 2021Instrumental Variable Value Iteration for Causal Offline Reinforcement Learning19 Feb
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning11 Feb

Other references · 19

  1. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  2. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  3. 2018The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimizationcited by 101
  4. 2018Confounding-Robust Policy Improvementcited by 76
  5. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  6. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  7. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  8. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  9. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 41
  10. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  11. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  12. 2016Gradient Descent Converges to Minimizerscited by 25
7 more
  1. 2020Statistically Efficient Off-Policy Policy Gradientscited by 25
  2. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 21
  3. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  4. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  5. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  6. 2022Model-Free and Model-Based Policy Evaluation when Causality is Uncertaincited by 12
  7. 2022Learning Mixtures of Markov Chains and MDPscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 2.

12 papers.

  1. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 7
  2. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 6
  3. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coveragecited by 5
  4. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  5. 2023Offline Recommender System Evaluation under Unobserved Confoundingcited by 4
  6. 2024Why Online Reinforcement Learning is Causalcited by 1
  7. 2025Off-Policy Evaluation for Sequential Persuasion Process with Unobserved Confoundingcited by 1
  8. 2025Model-Based Reinforcement Learning Under Confoundingcited by 1
  9. 2025Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applicationscited by 1
  10. 2024Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datacited by 0
  11. 2026Non-Parametric Rehearsal Learning via Conditional Mean Embeddingscited by 0
  12. 2026Robust Counterfactual Policy Optimisation via Nondeterministic Causal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confounding
  2. 2026Learning Optimal and Sample-Efficient Decision Policies with Guarantees
  3. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  4. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confounding
  5. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  8. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data