# Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/2607.08444 · HTML: https://mlsanity.com/paper/2607.08444/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 59 references that are in the map.
- First posted on arXiv 9 Jul 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 7 of 7)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [An Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2301.04462/) (2023)
- [Quantile QT-Opt for Risk-Aware Vision-Based Robotic Grasping](https://mlsanity.com/paper/1910.02787/) (2019)
- [Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Data](https://mlsanity.com/paper/2302.07690/) (2023)
- [Distributional Offline Policy Evaluation with Predictive Error Guarantees](https://mlsanity.com/paper/2302.09456/) (2023)
- [GAN Q-learning](https://mlsanity.com/paper/1805.04874/) (2018)
- [A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximation](https://mlsanity.com/paper/2502.14172/) (2025)

## Built on by (most cited first, 3 of 3)

- [Online Inference for Quantile Temporal Difference Learning in Distributional Reinforcement Learning](https://mlsanity.com/paper/2608.12973/) (2026)
- [A Finite-Sample Analysis of Quantile Temporal-Difference Learning](https://mlsanity.com/paper/2608.27313/) (2026)
- [Tail-Influence Sampling for CVaR Policy Evaluation](https://mlsanity.com/paper/2609.38096/) (2026)

## Abstract (arXiv)

In this paper, we study quantile-based distributional reinforcement learning from the perspective of statistical efficiency. We focus on distributional policy evaluation, whose goal is to characterize the return distribution, namely the distribution of discounted cumulative rewards under a given policy. To obtain a finite-dimensional representation of the return distribution, we consider the quantile fixed point ηm induced by the quantile-projected distributional Bellman equation. Assuming access to a generative model, we construct an estimator ηm(n) based on an empirical Markov decision process. For a fixed number of quantiles m, we establish a non-asymptotic error bound for ηm(n) and ηm under the supremum W_∞ metric, showing that the estimation error scales as widetildeO(√m/n) with respect to m and n. This implies that the quantile-based distributional policy evaluation problem can be solved with sample efficiency, achieving the optimal parametric √n convergence rate. We derive the asymptotic distribution of the quantile parameters √n(θm(n)-θm) and characterize the semiparametric efficiency bound, which is attained by our estimator. Beyond the fixed-dimensional setting, we investigate the asymptotic regime in which the number of quantiles diverges. We characterize the limit covariance structure and show that it matches the semiparametric efficiency bound of the nonparametric model for distributional policy evaluation, showing that quantile-based estimators remain asymptotically efficient in the infinite-dimensional limit. Finally, we establish a Berry--Esseen theorem for smooth functionals √n(ηm(n)(s)-ηm(s))f, thereby providing a foundation for statistically valid inference on functionals of the quantile-projected return distribution.