mlsanityData to 7 Oct 2026

Online Bootstrap Inference For Policy Evaluation in Reinforcement Learning

arXiv 2108.03706 ↗First posted 8 Aug 2021stat.ML cs.AI cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

The recent emergence of reinforcement learning has created a demand for robust statistical inference methods for the parameter estimates computed using these algorithms. Existing methods for statistical inference in online learning are restricted to settings involving independently sampled observations, while existing statistical inference methods in reinforcement learning (RL) are limited to the batch setting. The online bootstrap is a flexible and efficient approach for statistical inference in linear stochastic approximation algorithms, but its efficacy in settings involving Markov noise, such as RL, has yet to be explored. In this paper, we study the use of the online bootstrap method for statistical inference in RL. In particular, we focus on the temporal difference (TD) learning and Gradient TD (GTD) learning algorithms, which are themselves special instances of linear stochastic approximation under Markov noise. The method is shown to be distributionally consistent for statistical inference in policy evaluation, and numerical experiments are included to demonstrate the effectiveness of this algorithm at statistical inference tasks across a range of real RL environments.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 31 of its 86 references that are in the map. First posted on arXiv 8 Aug 2021.

Builds on

31 of 86 references.

References · 31

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2016OpenAI Gymcited by 1,684
  4. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  5. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  6. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  7. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  8. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  9. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  10. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 114
  11. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  12. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
19 more
  1. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  2. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  3. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  4. 2020On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentrationcited by 65
  5. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  6. 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
  7. 2020Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noisecited by 52
  8. 2019Finite-Time Performance Bounds and Adaptive Learning Rate Selection for Two Time-Scale Reinforcement Learningcited by 40
  9. 2019Inference and Uncertainty Quantification for Noisy Matrix Completioncited by 39
  10. 2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
  11. 2019Bootstrapping Upper Confidence Boundcited by 36
  12. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  13. 2020Reanalysis of Variance Reduced Temporal Difference Learningcited by 23
  14. 2021On the Stability of Random Matrix Product with Markovian Noise: Application to Linear Stochastic Approximation and TD Learningcited by 22
  15. 2017Statistical inference using SGDcited by 21
  16. 2020Steady State Analysis of Episodic Reinforcement Learningcited by 21
  17. 2019Characterizing the Exact Behaviors of Temporal Difference Learning Algorithms Using Markov Jump Linear System Theorycited by 17
  18. 2020Statistical Inference for Online Decision Making via Stochastic Gradient Descentcited by 17
  19. 2020Residual Bootstrap Exploration for Bandit Algorithmscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 8; 2024: 5; 2025: 6; 2026: 2.

26 papers.

Sort:

By area:

  1. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  2. 2024Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learningcited by 19
  3. 2023Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datacited by 18
  4. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  5. 2024High Confidence Level Inference is Almost Free using Parallel Stochastic Optimizationcited by 9
  6. 2022Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descentcited by 7
  7. 2023Online covariance estimation for stochastic gradient descent under Markovian samplingcited by 7
  8. 2023Statistical Inference with Stochastic Gradient Methods under φ-mixing Datacited by 6
  9. 2023Did we personalize? Assessing personalization by an online reinforcement learning algorithm using resamplingcited by 6
  10. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  11. 2024Advancing Retail Data Science: Comprehensive Evaluation of Synthetic Datacited by 5
  12. 2021Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learningcited by 4
14 more
  1. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  2. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  3. 2024Asymptotic Analysis of Sample-averaged Q-learningcited by 2
  4. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  5. 2025CLT and Edgeworth Expansion for m-out-of-n Bootstrap Estimators of The Studentized Mediancited by 2
  6. 2025Beyond Sin-Squared Error: Linear-Time Entrywise Uncertainty Quantification for Streaming PCAcited by 2
  7. 2025A Two-armed Bandit Framework for A/B Testingcited by 2
  8. 2025Improved Central Limit Theorem and Bootstrap Approximations for Linear Stochastic Approximationcited by 2
  9. 2024Incentivized Exploration with Stochastic Covariates: A Two-Stage Mechanism Design for Recommender Systemcited by 1
  10. 2023Deep Spectral Q-learning with Application to Mobile Healthcited by 0
  11. 2023An Online Bootstrap for Time Seriescited by 0
  12. 2025Generalized Fitted Q-Iteration with Clustered Datacited by 0
  13. 2026Model-based Bootstrap of Controlled Markov Chainscited by 0
  14. 2026Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learningcited by 0