mlsanityData to 14 Oct 2026

Deeply-Debiased Off-Policy Interval Estimation

arXiv 2105.04646 ↗First posted 10 May 2021stat.ML cs.AI cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Off-policy evaluation learns a target policy's value with a historical dataset generated by a different behavior policy. In addition to a point estimate, many applications would benefit significantly from having a confidence interval (CI) that quantifies the uncertainty of the point estimate. In this paper, we propose a novel deeply-debiasing procedure to construct an efficient, robust, and flexible CI on a target policy's value. Our method is justified by theoretical results and numerical experiments. A Python implementation of the proposed procedure is available at https://github.com/RunzheStat/D2OPE.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 31 of its 62 references that are in the map. First posted on arXiv 10 May 2021.

Builds on

31 of 62 references.

References · 31

  1. 2016OpenAI Gymcited by 1,736
  2. 2018Learning Dexterous In-Hand Manipulationcited by 687
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  4. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  8. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  9. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  10. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  11. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  12. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
19 more
  1. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  2. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  3. 2019Confidence Intervals for Policy Evaluation in Adaptive Experimentscited by 95
  4. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  5. 2017Double/Debiased/Neyman Machine Learning of Treatment Effectscited by 76
  6. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  7. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  8. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  9. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  10. 2020Inference for Batched Banditscited by 63
  11. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  12. 2017Accurate Inference for Adaptive Linear Modelscited by 39
  13. 2020Adaptive Estimator Selection for Off-Policy Evaluationcited by 38
  14. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  15. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  16. 2017Orthogonal Machine Learning: Power and Limitationscited by 28
  17. 2017Randomized incomplete U-statistics in high dimensionscited by 12
  18. 2020Testing Mediation Effects Using Logic of Boolean Matricescited by 6
  19. 2016Evaluating the Impact of Treating the Optimal Subgroupcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 10; 2024: 3; 2025: 7; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  3. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  4. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  5. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  6. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  7. 2022Safe Exploration for Efficient Policy Evaluation and Comparisoncited by 15
  8. 2022Conformal Off-policy Predictioncited by 14
  9. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  10. 2025Doubly Robust Alignment for Large Language Modelscited by 12
  11. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  12. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
24 more
  1. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  2. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  3. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  4. 2023Conformal Off-Policy Evaluation in Markov Decision Processescited by 6
  5. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  6. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  7. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  8. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  9. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  10. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  11. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 4
  12. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3
  13. 2022On the Reuse Bias in Off-Policy Reinforcement Learningcited by 2
  14. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  15. 2025Off-Policy Evaluation Under Nonignorable Missing Datacited by 2
  16. 2021Dynamic Selection in Algorithmic Decision-makingcited by 1
  17. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  18. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  19. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  20. 2023Minimax Weight Learning for Absorbing MDPscited by 0
  21. 2023Asymptotic Inference for Multi-Stage Stationary Treatment Policy with Variable Selectioncited by 0
  22. 2025An Orthogonal Learner for Individualized Outcomes in Markov Decision Processescited by 0
  23. 2025Scalable Offline Metrics for Autonomous Drivingcited by 0
  24. 2026Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learningcited by 0