Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Not on the map. Why?
Abstract (arXiv)
Distributional reinforcement learning algorithms have attempted to utilize estimated uncertainty for exploration, such as optimism in the face of uncertainty. However, using the estimated variance for optimistic exploration may cause biased data collection and hinder convergence or performance. In this paper, we present a novel distributional reinforcement learning algorithm that selects actions by randomizing risk criterion to avoid one-sided tendency on risk. We provide a perturbed distributional Bellman optimality operator by distorting the risk measure and prove the convergence and optimality of the proposed method with the weaker contraction property. Our theoretical results support that the proposed method does not fall into biased exploration and is guaranteed to converge to an optimal return. Finally, we empirically show that our method outperforms other existing distribution-based algorithms in various environments including Atari 55 games.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 25 Oct 2023.
Builds on
29 of 41 references.
References · 29
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2016Deep Exploration via Bootstrapped DQNcited by 527
- 2018Is Q-learning Provably Efficient?cited by 487
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
- 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
- 2015Data-driven Distributionally Robust Optimization Using the Wasserstein Metric: Performance Guarantees and Tractable Reformulationscited by 291
- 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
- 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
- 2017Deep Exploration via Randomized Value Functionscited by 156
- 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
17 more
- 2014Algorithms for CVaR Optimization in MDPscited by 99
- 2019Better Exploration with Optimistic Actor-Criticcited by 88
- 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
- 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
- 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
- 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
- 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
- 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
- 2019Distributionally Robust Reinforcement Learningcited by 42
- 2018Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approachcited by 29
- 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 29
- 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
- 2021Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learningcited by 15
- 2018Exploration by Distributional Reinforcement Learningcited by 13
- 2018QUOTA: The Quantile Option Architecture for Reinforcement Learningcited by 13
- 2022Risk Perspective Exploration in Distributional Reinforcement Learningcited by 5
- 2018The Potential of the Return Distribution for Exploration in RLcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 2.
3 papers.