mlsanityData to 7 Oct 2026

Bootstrapping Fitted Q-Evaluation for Off-Policy Inference

arXiv 2102.03607 ↗First posted 6 Feb 2021stat.ML cs.LG math.STcited by 40

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Bootstrapping provides a flexible and effective approach for assessing the quality of batch reinforcement learning, yet its theoretical property is less understood. In this paper, we study the use of bootstrapping in off-policy evaluation (OPE), and in particular, we focus on the fitted Q-evaluation (FQE) that is known to be minimax-optimal in the tabular and linear-model cases. We propose a bootstrapping FQE method for inferring the distribution of the policy evaluation error and show that this method is asymptotically efficient and distributionally consistent for off-policy statistical inference. To overcome the computation limit of bootstrapping, we further adapt a subsampling procedure that improves the runtime by an order of magnitude. We numerically evaluate the bootrapping method in classical RL environments for confidence interval estimation, estimating the variance of off-policy evaluator, and estimating the correlation between multiple off-policy evaluators.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 49 references that are in the map. First posted on arXiv 6 Feb 2021.

Builds on

27 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient8 Nov
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  3. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul

Other references · 24

  1. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2019Batch Policy Learning under Constraintscited by 242
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  6. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  7. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  8. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  9. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  10. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  11. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 118
  12. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
12 more
  1. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  2. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  3. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  4. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  5. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  6. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  7. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  8. 2019Bootstrapping Upper Confidence Boundcited by 36
  9. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  10. 2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28
  11. 2015A subsampled double bootstrap for massive datacited by 5
  12. 2017Bootstrapping for multivariate linear regression modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 12; 2024: 9; 2025: 6; 2026: 5.

40 papers.

Sort:

By area:

  1. 2022Policy-Adaptive Estimator Selection for Off-Policy Evaluationcited by 24
  2. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  3. 2023Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datacited by 18
  4. 2022Conformal Off-policy Predictioncited by 14
  5. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  6. 2022Sample Complexity of Nonparametric Off-Policy Evaluation on Low-Dimensional Manifolds using Deep Networkscited by 12
  7. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  8. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  9. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  10. 2023Variational Latent Branching Model for Off-Policy Evaluationcited by 6
  11. 2023Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resamplingcited by 6
  12. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
28 more
  1. 2023Off-Policy Evaluation for Human Feedbackcited by 6
  2. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  3. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  4. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  5. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  6. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  7. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samplescited by 4
  8. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  9. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  10. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  11. 2022Policy Optimization with Sparse Global Contrastive Explanationscited by 2
  12. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  13. 2023K-Nearest-Neighbor Resampling for Off-Policy Evaluation in Stochastic Controlcited by 2
  14. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 2
  15. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  16. 2024Markov Chain Variance Estimation: A Stochastic Approximation Approachcited by 2
  17. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  18. 2022Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iterationcited by 1
  19. 2024Why long model-based rollouts are no reason for bad Q-value estimatescited by 1
  20. 2025Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluationcited by 1
  21. 2025Central Limit Theorems for Transition Probabilities of Controlled Markov Chainscited by 1
  22. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theorycited by 1
  23. 2026Designing Time Series Experiments in A/B Testing with Transformer Reinforcement Learningcited by 1
  24. 2026The hidden risks of temporal resampling in clinical reinforcement learningcited by 1
  25. 2026Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learningcited by 1
  26. 2024Off-Policy Selection for Initiating Human-Centric Experimental Designcited by 0
  27. 2026Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluationcited by 0
  28. 2026Model-based Bootstrap of Controlled Markov Chainscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning
  2. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  3. 2019Combining Parametric and Nonparametric Models for Off-Policy Evaluation
  4. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2023The In-Sample Softmax for Offline Reinforcement Learning
  7. 2021Offline RL Without Off-Policy Evaluation
  8. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation