Factors of Influence of the Overestimation Bias of Q-Learning
Not on the map. Why?
Abstract (arXiv)
We study whether the learning rate α, the discount factor γ and the reward signal r have an influence on the overestimation bias of the Q-Learning algorithm. Our preliminary results in environments which are stochastic and that require the use of neural networks as function approximators, show that all three parameters influence overestimation significantly. By carefully tuning α and γ, and by using an exponential moving average of r in Q-Learning's temporal difference target, we show that the algorithm can learn value estimates that are more accurate than the ones of several other popular model-free methods that have addressed its overestimation bias in the past.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 32 references that are in the map. First posted on arXiv 11 Oct 2022.
Builds on
13 of 32 references.
References · 13
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2016OpenAI Gymcited by 1,684
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
- 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
- 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
- 2019Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approachcited by 32
- 2021Ensemble Bootstrapping for Q-Learningcited by 30
- 2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 26
- 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
- 2020Self-correcting Q-Learningcited by 9
- 2020Deep Reinforcement Learning with Weighted Q-Learningcited by 6
Built on by
Not cited yet.