mlsanityData to 14 Oct 2026

Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settings

arXiv 2001.04515 ↗First posted 13 Jan 2020stat.ML cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning is a general technique that allows an agent to learn an optimal policy and interact with an environment in sequential decision making problems. The goodness of a policy is measured by its value function starting from some initial state. The focus of this paper is to construct confidence intervals (CIs) for a policy's value in infinite horizon settings where the number of decision points diverges to infinity. We propose to model the action-value state function (Q-function) associated with a policy based on series/sieve method to derive its confidence interval. When the target policy depends on the observed data as well, we propose a SequentiAl Value Evaluation (SAVE) method to recursively update the estimated policy and its value estimator. As long as either the number of trajectories or the number of decision points diverges to infinity, we show that the proposed CI achieves nominal coverage even in cases where the optimal policy is not unique. Simulation studies are conducted to back up our theoretical findings. We apply the proposed method to a dataset from mobile health studies and find that reinforcement learning algorithms could help improve patient's health status. A Python implementation of the proposed procedure is available at https://github.com/shengzhang37/SAVE.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 7; 2022: 18; 2023: 20; 2024: 7; 2025: 10; 2026: 2.

65 papers.

Sort:

By area:

  1. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  2. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  3. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  4. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  5. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  6. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  7. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  8. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  9. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 21
  10. 2025AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guaranteescited by 21
  11. 2022An Instrumental Variable Approach to Confounded Off-Policy Evaluationcited by 20
  12. 2022Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processescited by 19
38 more
  1. 2022Off-Policy Fitted Q-Evaluation with Differentiable Function Approximators: Z-Estimation and Inference Theorycited by 18
  2. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  3. 2022Conformal Off-policy Predictioncited by 14
  4. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  5. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  6. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 10
  7. 2023Testing for the Markov Property in Time Series via Deep Conditional Generative Learningcited by 9
  8. 2020Efficient Evaluation of Natural Stochastic Policies in Offline Reinforcement Learningcited by 8
  9. 2022Policy Evaluation for Temporal and/or Spatial Dependent Experimentscited by 8
  10. 2022Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descentcited by 8
  11. 2023Estimation and Inference in Distributional Reinforcement Learningcited by 8
  12. 2022Reinforcement Learning with Heterogeneous Data: Estimation and Inferencecited by 7
  13. 2023Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimizationcited by 7
  14. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7
  15. 2023Statistical Inference with Stochastic Gradient Methods under φ-mixing Datacited by 7
  16. 2023Sequential Knockoffs for Variable Selection in Reinforcement Learningcited by 7
  17. 2023Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharingcited by 7
  18. 2023Off-policy Evaluation in Doubly Inhomogeneous Environmentscited by 7
  19. 2023Conformal Off-Policy Evaluation in Markov Decision Processescited by 6
  20. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  21. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  22. 2024Combining Experimental and Historical Data for Policy Evaluationcited by 6
  23. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  24. 2023Revisiting Bellman Errors for Offline Model Selectioncited by 5
  25. 2024Two-way Deconfounder for Off-policy Evaluation in Causal Reinforcement Learningcited by 5
  26. 2025Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessingcited by 5
  27. 2025Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approachcited by 5
  28. 2022A Multi-Agent Reinforcement Learning Framework for Off-Policy Evaluation in Two-sided Marketscited by 4
  29. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  30. 2024Off-policy Evaluation with Deeply-abstracted Statescited by 4
  31. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learningcited by 3
  32. 2022Doubly Inhomogeneous Reinforcement Learningcited by 3
  33. 2023STEEL: Singularity-aware Reinforcement Learningcited by 3
  34. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  35. 2023A Reinforcement Learning Framework for Dynamic Mediation Analysiscited by 2
  36. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  37. 2025Statistical Inference in Reinforcement Learning: A Selective Surveycited by 2
  38. 2025Off-Policy Evaluation Under Nonignorable Missing Datacited by 2

50 most cited of 65 All 65 (JSON)