mlsanityData to 14 Oct 2026

Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Framework

arXiv 2309.13278 ↗First posted 23 Sep 2023stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We study high-confidence off-policy evaluation in the context of infinite-horizon Markov decision processes, where the objective is to establish a confidence interval (CI) for the target policy value using only offline data pre-collected from unknown behavior policies. This task faces two primary challenges: providing a comprehensive and rigorous error quantification in CI estimation, and addressing the distributional shift that results from discrepancies between the distribution induced by the target policy and the offline data-generating process. Motivated by an innovative unified error analysis, we jointly quantify the two sources of estimation errors: the misspecification error on modeling marginalized importance weights and the statistical uncertainty due to sampling, within a single interval. This unified framework reveals a previously hidden tradeoff between the errors, which undermines the tightness of the CI. Relying on a carefully designed discriminator function, the proposed estimator achieves a dual purpose: breaking the curse of the tradeoff to attain the tightest possible CI, and adapting the CI to ensure robustness against distributional shifts. Our method is applicable to time-dependent data without assuming any weak dependence conditions via leveraging a local supermartingale/martingale structure. Theoretically, we show that our algorithm is sample-efficient, error-robust, and provably convergent even in non-linear function approximation settings. The numerical performance of the proposed method is examined in synthetic datasets and an OhioT1DM mobile health study.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 51 of its 75 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

51 of 75 references.

References · 51

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2016OpenAI Gymcited by 1,736
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  8. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  11. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  12. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
38 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  2. 2019Batch Policy Learning under Constraintscited by 244
  3. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  4. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  5. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  6. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 145
  7. 2019A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approachcited by 143
  8. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  9. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  10. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  11. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 119
  12. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  13. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  14. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  15. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  16. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  17. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  18. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  19. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiencycited by 56
  20. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  21. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  22. 2021Oracle Complexity in Nonsmooth Nonconvex Optimizationcited by 39
  23. 2019A Kernel Loss for Solving the Bellman Equationcited by 37
  24. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  25. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  26. 2021A Minimax Learning Approach to Off-Policy Evaluation in Confounded Partially Observable Markov Decision Processescited by 36
  27. 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
  28. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  29. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  30. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  31. 2019Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learningcited by 28
  32. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  33. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  34. 2014Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spacescited by 19
  35. 2017Boosting the Actor with Dual Criticcited by 14
  36. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  37. 2020Causal Effect Estimation and Optimal Dose Suggestions in Mobile Healthcited by 8
  38. 2020Reinforcement Learning for Strategic Recommendationscited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 4.

6 papers.

  1. 2023Bi-Level Offline Policy Optimization with Limited Explorationcited by 5
  2. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  3. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  4. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  5. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  6. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0