mlsanityData to 14 Oct 2026

Model-Free and Model-Based Policy Evaluation when Causality is Uncertain

arXiv 2204.00956 ↗First posted 2 Apr 2022cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

When decision-makers can directly intervene, policy evaluation algorithms give valid causal estimates. In off-policy evaluation (OPE), there may exist unobserved variables that both impact the dynamics and are used by the unknown behavior policy. These "confounders" will introduce spurious correlations and naive estimates for a new policy will be biased. We develop worst-case bounds to assess sensitivity to these unobserved confounders in finite horizons when confounders are drawn iid each period. We demonstrate that a model-based approach with robust MDPs gives sharper lower bounds by exploiting domain knowledge about the dynamics. Finally, we show that when unobserved confounders are persistent over time, OPE is far more difficult and existing techniques produce extremely conservative bounds.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 21 of its 39 references that are in the map. First posted on arXiv 2 Apr 2022.

Builds on

21 of 39 references.

References · 21

  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  3. 2019Batch Policy Learning under Constraintscited by 244
  4. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  5. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  6. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  7. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  8. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  9. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  10. 2018Confounding-Robust Policy Improvementcited by 76
  11. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  12. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
9 more
  1. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 61
  2. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 57
  3. 2018Interval Estimation of Individual-Level Causal Effects Under Unobserved Confoundingcited by 51
  4. 2018Bounds on the conditional and average treatment effect with unobserved confounding factorscited by 39
  5. 2018Importance Sampling Policy Evaluation with an Estimated Behavior Policycited by 34
  6. 2018Flexible sensitivity analysis for observational studies without observable implicationscited by 18
  7. 2019Combining Parametric and Nonparametric Models for Off-Policy Evaluationcited by 17
  8. 2019Policy Evaluation with Latent Confounders via Optimal Balancecited by 8
  9. 2018Algorithmic Decision Making in the Presence of Unmeasured Confoundingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 2; 2025: 2; 2026: 2.

12 papers.

  1. 2023A Survey on Causal Reinforcement Learningcited by 45
  2. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  3. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 13
  4. 2022Learning Mixtures of Markov Chains and MDPscited by 12
  5. 2022Offline Policy Evaluation and Optimization under Confoundingcited by 12
  6. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
  7. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  8. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  9. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  10. 2024Why Online Reinforcement Learning is Causalcited by 1
  11. 2026Non-Parametric Rehearsal Learning via Conditional Mean Embeddingscited by 0
  12. 2026Robust Counterfactual Policy Optimisation via Nondeterministic Causal Modelscited by 0