# Uncertainty Quantification and Exploration for Reinforcement Learning

arXiv: https://arxiv.org/abs/1910.05471 · HTML: https://mlsanity.com/paper/1910.05471/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2019.

## Built on by (most cited first, 11 of 11)

- [Estimation and Inference in Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.17262/) (2023)
- [Failure Identification in Imitation Learning Via Statistical and Semantic Filtering](https://mlsanity.com/paper/2604.13788/) (2026)
- [Advancing Investment Frontiers: Industry-grade Deep Reinforcement Learning for Portfolio Optimization](https://mlsanity.com/paper/2403.07916/) (2024)
- [Auto-exploration for online reinforcement learning](https://mlsanity.com/paper/2512.06244/) (2025)
- [Sample-Efficient "Clustering and Conquer" Procedures for Parallel Large-Scale Ranking and Selection](https://mlsanity.com/paper/2402.02196/) (2024)
- [Central Limit Theorems for Transition Probabilities of Controlled Markov Chains](https://mlsanity.com/paper/2508.01517/) (2025)
- [Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationarity](https://mlsanity.com/paper/2605.03393/) (2026)
- [Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning](https://mlsanity.com/paper/2608.12973/) (2026)
- [Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitions](https://mlsanity.com/paper/2601.18107/) (2026)
- [Model-based Bootstrap of Controlled Markov Chains](https://mlsanity.com/paper/2605.12410/) (2026)
- [Tail-Influence Sampling for CVaR Policy Evaluation](https://mlsanity.com/paper/2609.38096/) (2026)

## Abstract (arXiv)

We investigate statistical uncertainty quantification for reinforcement learning (RL) and its implications in exploration policy. Despite ever-growing literature on RL applications, fundamental questions about inference and error quantification, such as large-sample behaviors, appear to remain quite open. In this paper, we fill in the literature gap by studying the central limit theorem behaviors of estimated Q-values and value functions under various RL settings. In particular, we explicitly identify closed-form expressions of the asymptotic variances, which allow us to efficiently construct asymptotically valid confidence regions for key RL quantities. Furthermore, we utilize these asymptotic expressions to design an effective exploration strategy, which we call Q-value-based Optimal Computing Budget Allocation (Q-OCBA). The policy relies on maximizing the relative discrepancies among the Q-value estimates. Numerical experiments show superior performances of our exploration strategy than other benchmark policies.