mlsanityData to 7 Oct 2026

Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process

arXiv 2202.10589 ↗First posted 22 Feb 2022stat.ML cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

This paper is concerned with constructing a confidence interval for a target policy's value offline based on a pre-collected observational data in infinite horizon settings. Most of the existing works assume no unmeasured variables exist that confound the observed actions. This assumption, however, is likely to be violated in real applications such as healthcare and technological industries. In this paper, we show that with some auxiliary variables that mediate the effect of actions on the system dynamics, the target policy's value is identifiable in a confounded Markov decision process. Based on this result, we develop an efficient off-policy value estimator that is robust to potential model misspecification and provide rigorous uncertainty quantification. Our method is justified by theoretical results, simulated and real datasets obtained from ridesharing companies. A Python implementation of the proposed procedure is available at https://github.com/Mamba413/cope.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 38 of its 85 references that are in the map. First posted on arXiv 22 Feb 2022.

Builds on

38 of 85 references.

References · 38

  1. 2015Estimation and Inference of Heterogeneous Treatment Effects using Random Forestscited by 597
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  4. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 242
  7. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  8. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2020An Introduction to Proximal Causal Learningcited by 91
  11. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  12. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
26 more
  1. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  2. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 65
  3. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  4. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  5. 2020Off-policy Policy Evaluation For Sequential Decisions Under Unobserved Confoundingcited by 60
  6. 2020Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learningcited by 56
  7. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 54
  8. 2021Causal Inference Under Unmeasured Confounding With Negative Controls: A Minimax Learning Approachcited by 43
  9. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  10. 2020Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacited by 40
  11. 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
  12. 2021Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processescited by 40
  13. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  14. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  15. 2020Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inferencecited by 34
  16. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 34
  17. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  18. 2018Fair Algorithms for Learning in Allocation Problemscited by 29
  19. 2021A Spectral Approach to Off-Policy Evaluation for POMDPscited by 20
  20. 2020Personalized Policy Learning using Longitudinal Mobile Health Datacited by 10
  21. 2020Latent-state models for precision medicinecited by 6
  22. 2016Evaluating the Impact of Treating the Optimal Subgroupcited by 5
  23. 2014Constructing Dynamic Treatment Regimes in Infinite-Horizon Settingscited by 4
  24. 2020Robust Batch Policy Learning in Markov Decision Processescited by 4
  25. 2019Resampling-based Confidence Intervals for Model-free Robust Inference on Optimal Treatment Regimescited by 3
  26. 2021Causal mediation analysis with mediator values below an assay limitcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 8; 2024: 3; 2025: 7; 2026: 5.

31 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 92
  2. 2023A Survey on Causal Reinforcement Learningcited by 43
  3. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  4. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  5. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
  6. 2022Off-Policy Evaluation for Episodic Partially Observable Markov Decision Processes under Non-Parametric Modelscited by 18
  7. 2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
  8. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  9. 2024Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedbackcited by 10
  10. 2022Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descentcited by 7
  11. 2023Delphic Offline Reinforcement Learning under Nonidentifiable Hidden Confoundingcited by 7
  12. 2023Statistical Inference with Stochastic Gradient Methods under φ-mixing Datacited by 6
19 more
  1. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  2. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  3. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  4. 2025Efficient and Sharp Off-Policy Learning under Unobserved Confoundingcited by 5
  5. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  6. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  7. 2022Blessing from Human-AI Interaction: Super Reinforcement Learning in Confounded Environmentscited by 3
  8. 2023Contextual Dynamic Pricing with Strategic Buyerscited by 3
  9. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  10. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  11. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  12. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  13. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  14. 2022Offline Reinforcement Learning for Human-Guided Human-Machine Interaction with Private Informationcited by 0
  15. 2023A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learningcited by 0
  16. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  17. 2026Causal Imitation Learning Under Measurement Error and Distribution Shiftcited by 0
  18. 2026Learning Optimal Individualized Decision Rules with Conditional Demographic Paritycited by 0
  19. 2026Robust Sequential Experimental Design for A/B Testingcited by 0