# Factors of Influence of the Overestimation Bias of Q-Learning

arXiv: https://arxiv.org/abs/2210.05262 · HTML: https://mlsanity.com/paper/2210.05262/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 13 of 13)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Maxmin Q-learning: Controlling the Estimation Bias of Q-learning](https://mlsanity.com/paper/2002.06487/) (2020)
- [Hyperbolic Discounting and Learning over Multiple Horizons](https://mlsanity.com/paper/1902.06865/) (2019)
- [Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach](https://mlsanity.com/paper/1902.02893/) (2019)
- [Ensemble Bootstrapping for Q-Learning](https://mlsanity.com/paper/2103.00445/) (2021)
- [How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies](https://mlsanity.com/paper/1512.02011/) (2015)
- [Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning](https://mlsanity.com/paper/1906.00572/) (2019)
- [Self-correcting Q-Learning](https://mlsanity.com/paper/2012.01100/) (2020)
- [Deep Reinforcement Learning with Weighted Q-Learning](https://mlsanity.com/paper/2003.09280/) (2020)
- [Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets](https://mlsanity.com/paper/2111.02787/) (2021)

## Abstract (arXiv)

We study whether the learning rate α, the discount factor γ and the reward signal r have an influence on the overestimation bias of the Q-Learning algorithm. Our preliminary results in environments which are stochastic and that require the use of neural networks as function approximators, show that all three parameters influence overestimation significantly. By carefully tuning α and γ, and by using an exponential moving average of r in Q-Learning's temporal difference target, we show that the algorithm can learn value estimates that are more accurate than the ones of several other popular model-free methods that have addressed its overestimation bias in the past.