Estimation and Inference in Distributional Reinforcement Learning

In this paper, we study distributional reinforcement learning from the perspective of statistical efficiency. We investigate distributional policy evaluation, aiming to estimate the complete return distribution (denoted ηπ) attained by a given policy π. We use the certainty-equivalence method to construct our estimator hatηπ, given a generative model is available. In this circumstance we need a dataset of size widetilde Oleft(frac|S||A|varepsilon2p(1-γ)2p+2right) to guarantee the p-Wasserstein metric between hatηπ and ηπ less than varepsilon with high probability. This implies the distributional policy evaluation problem can be solved with sample efficiency. Also, we show that under different mild assumptions a dataset of size widetilde Oleft(frac|S||A|varepsilon2(1-γ)4right) suffices to ensure the Kolmogorov metric and total variation metric between hatηπ and ηπ is below varepsilon with high probability. Furthermore, we investigate the asymptotic behavior of hatηπ. We demonstrate that the ``empirical process'' √n(hatηπ-ηπ) converges weakly to a Gaussian process in the space of bounded functionals on Lipschitz function class ℓ^∞(FW), also in the space of bounded functionals on indicator function class ℓ^∞(FKS) and bounded measurable function class ℓ^∞(FTV) when some mild conditions hold. Our findings give rise to a unified approach to statistical inference of a wide class of statistical functionals of ηπ.