mlsanityData to 7 Oct 2026

Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation

arXiv 2007.13609 ↗First posted 27 Jul 2020cs.LG stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, it is typical to use the empirically observed transitions and rewards to estimate the value of a policy via either model-based or Q-fitting approaches. Although straightforward, these techniques in general yield biased estimates of the true value of the policy. In this work, we investigate the potential for statistical bootstrapping to be used as a way to take these biased estimates and produce calibrated confidence intervals for the true value of the policy. We identify conditions - specifically, sufficient data size and sufficient coverage - under which statistical bootstrapping in this setting is guaranteed to yield correct confidence intervals. In practical situations, these conditions often do not hold, and so we discuss and propose mechanisms that can be employed to mitigate their effects. We evaluate our proposed method and show that it can yield accurate confidence intervals in a variety of conditions, including challenging continuous control environments and small data regimes.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 5; 2023: 6; 2024: 1; 2025: 3; 2026: 3.

28 papers.

Sort:

By area:

  1. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  2. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  5. 2021Universal Off-Policy Evaluationcited by 43
  6. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  7. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 32
  8. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  9. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  10. 2023Hallucinated Adversarial Control for Conservative Offline Policy Evaluationcited by 13
  11. 2021Non-asymptotic Confidence Intervals of Off-policy Evaluation: Primal and Dual Boundscited by 11
  12. 2021High-Confidence Off-Policy (or Counterfactual) Variance Estimationcited by 8
16 more
  1. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  2. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  3. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  4. 2024Primal-Dual Spectral Representation for Off-policy Evaluationcited by 4
  5. 2025Trajectory World Models for Heterogeneous Environmentscited by 4
  6. 2021Supervised Off-Policy Rankingcited by 3
  7. 2021Debiasing Samples from Online Learning Using Bootstrapcited by 3
  8. 2022Hybrid Value Estimation for Off-policy Evaluation and Offline Reinforcement Learningcited by 3
  9. 2023Probabilistic Offline Policy Ranking with Approximate Bayesian Computationcited by 3
  10. 2025STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluationcited by 2
  11. 2022Offline Policy Comparison with Confidence: Benchmarks and Baselinescited by 0
  12. 2023Conservative Exploration for Policy Optimization via Off-Policy Policy Evaluationcited by 0
  13. 2025Fully Offline Reinforcement Learningcited by 0
  14. 2026Formulating Reinforcement Learning for Human-Robot Collaboration through Off-Policy Evaluationcited by 0
  15. 2026Model-based Bootstrap of Controlled Markov Chainscited by 0
  16. 2026Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learningcited by 0