mlsanityData to 14 Oct 2026

Statistical Inference in Reinforcement Learning: A Selective Survey

arXiv 2502.16195 ↗First posted 22 Feb 2025stat.ML cs.LGcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) is concerned with how intelligence agents take actions in a given environment to maximize the cumulative reward they receive. In healthcare, applying RL algorithms could assist patients in improving their health status. In ride-sharing platforms, applying RL algorithms could increase drivers' income and customer satisfaction. For large language models, applying RL algorithms could align their outputs with human preferences. Over the past decade, RL has been arguably one of the most vibrant research frontiers in machine learning. Nevertheless, statistics as a field, as opposed to computer science, has only recently begun to engage with RL both in depth and in breadth. This chapter presents a selective review of statistical inferential tools for RL, covering both hypothesis testing and confidence interval construction. Our goal is to highlight the value of statistical inference in RL for both the statistics and machine learning communities, and to promote the broader application of classical statistical inference tools in this vibrant area of research.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 51 of its 102 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

51 of 102 references.

References · 51

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  4. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  6. 2019Batch Policy Learning under Constraintscited by 244
  7. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  8. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  9. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  10. 2019Confidence Intervals for Policy Evaluation in Adaptive Experimentscited by 95
  11. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  12. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
38 more
  1. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  2. 2019Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activitycited by 70
  3. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  4. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  5. 2020Statistical Inference of the Value Function for Reinforcement Learning in Infinite Horizon Settingscited by 65
  6. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  7. 2020Inference for Batched Banditscited by 63
  8. 2019Off-Policy Estimation of Long-Term Average Outcomes with Applications to Mobile Healthcited by 42
  9. 2021Bootstrapping Fitted Q-Evaluation for Off-Policy Inferencecited by 40
  10. 2019Conditional Independence Testing using Generative Adversarial Networkscited by 36
  11. 2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
  12. 2021Deeply-Debiased Off-Policy Interval Estimationcited by 36
  13. 2021Post-Contextual-Bandit Inferencecited by 32
  14. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  15. 2020Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Makingcited by 30
  16. 2020Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Frameworkcited by 29
  17. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  18. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  19. 2020Concentration Inequalities for Statistical Inferencecited by 24
  20. 2021Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorabilitycited by 24
  21. 2020Double Generative Adversarial Networks for Conditional Independence Testingcited by 22
  22. 2021Online Multi-Armed Bandits with Adaptive Inferencecited by 22
  23. 2022Markovian Interference in Experimentscited by 22
  24. 2021Minimum-Delay Adaptation in Non-Stationary Reinforcement Learning via Online High-Confidence Change-Point Detectioncited by 18
  25. 2021Projected State-action Balancing Weights for Offline Reinforcement Learningcited by 16
  26. 2023A/B Testing: A Systematic Literature Reviewcited by 16
  27. 2020Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approachcited by 15
  28. 2022Testing Stationarity and Change Point Detection in Reinforcement Learningcited by 12
  29. 2023DRCFS: Doubly Robust Causal Feature Selectioncited by 12
  30. 2023Optimal Treatment Allocation for Efficient Policy Evaluation in Sequential Decision Makingcited by 11
  31. 2022Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizonscited by 10
  32. 2023Testing for the Markov Property in Time Series via Deep Conditional Generative Learningcited by 9
  33. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learningcited by 8
  34. 2024Doubly Robust Conditional Independence Testing with Generative Neural Networkscited by 7
  35. 2023Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Frameworkcited by 6
  36. 2024Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experimentscited by 6
  37. 2021Testing Directed Acyclic Graph via Structural, Supervised and Generative Adversarial Learningcited by 4
  38. 2025Testing Conditional Mean Independence Using Generative Neural Networkscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  2. 2025A Two-armed Bandit Framework for A/B Testingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Statistical and Algorithmic Foundations of Reinforcement Learning
  2. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions
  3. 2024Introduction to Reinforcement Learning
  4. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  7. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF
  8. 2023Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy Optimization