Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
Not on the map. Why?
Abstract (arXiv)
Distributional reinforcement learning improves performance by capturing environmental stochasticity, but a comprehensive theoretical understanding of its effectiveness remains elusive. In addition, the intractable element of the infinite dimensionality of distributions has been overlooked. In this paper, we present a regret analysis of distributional reinforcement learning with general value function approximation in a finite episodic Markov decision process setting. We first introduce a key notion of textitBellman unbiasedness which is essential for exactly learnable and provably efficient distributional updates in an online manner. Among all types of statistical functionals for representing infinite-dimensional return distributions, our theoretical results demonstrate that only moment functionals can exactly capture the statistical information. Secondly, we propose a provably efficient algorithm, textttSF-LSVI, that achieves a tight regret bound of tildeO(dE H^frac32√K) where H is the horizon, K is the number of episodes, and dE is the eluder dimension of a function class.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 31 Jul 2024.
Builds on
29 of 45 references.
References · 29
- 2017A Distributional Perspective on Reinforcement Learningcited by 743
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
- 2018Is Q-learning Provably Efficient?cited by 485
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
- 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
- 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
- 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
- 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
- 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 180
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
- 2017Deep Exploration via Randomized Value Functionscited by 155
17 more
- 2019Optimism in Reinforcement Learning with Generalized Linear Function Approximationcited by 111
- 2018On Oracle-Efficient PAC RL with Rich Observationscited by 71
- 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
- 2019Statistics and Samples in Distributional Reinforcement Learningcited by 62
- 2022Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescited by 57
- 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
- 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
- 2022Efficient Risk-Averse Reinforcement Learningcited by 41
- 2016On Lower Bounds for Regret in Reinforcement Learningcited by 39
- 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
- 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
- 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
- 2023Trust Region-Based Safe Distributional Reinforcement Learning for Multiple Constraintscited by 20
- 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
- 2023Beyond Average Return in Markov Decision Processescited by 11
- 2023Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterioncited by 3
- 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
Built on by
Not cited yet.