# Self-correcting Q-Learning

arXiv: https://arxiv.org/abs/2012.01100 · HTML: https://mlsanity.com/paper/2012.01100/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 9 of 9)

- [Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback](https://mlsanity.com/paper/2306.11918/) (2023)
- [On the Estimation Bias in Double Q-Learning](https://mlsanity.com/paper/2109.14419/) (2021)
- [Balanced Q-learning: Combining the Influence of Optimistic and Pessimistic Targets](https://mlsanity.com/paper/2111.02787/) (2021)
- [Smoothed Q-learning](https://mlsanity.com/paper/2303.08631/) (2023)
- [Use the Online Network If You Can: Towards Fast and Stable Reinforcement Learning](https://mlsanity.com/paper/2510.02590/) (2025)
- [Two-Step Q-Learning](https://mlsanity.com/paper/2407.02369/) (2024)
- [Deep Double Q-learning](https://mlsanity.com/paper/2507.00275/) (2025)
- [Factors of Influence of the Overestimation Bias of Q-Learning](https://mlsanity.com/paper/2210.05262/) (2022)
- [Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning](https://mlsanity.com/paper/2409.06356/) (2024)

## Abstract (arXiv)

The Q-learning algorithm is known to be affected by the maximization bias, i.e. the systematic overestimation of action values, an important issue that has recently received renewed attention. Double Q-learning has been proposed as an efficient algorithm to mitigate this bias. However, this comes at the price of an underestimation of action values, in addition to increased memory requirements and a slower convergence. In this paper, we introduce a new way to address the maximization bias in the form of a "self-correcting algorithm" for approximating the maximum of an expected value. Our method balances the overestimation of the single estimator used in conventional Q-learning and the underestimation of the double estimator used in Double Q-learning. Applying this strategy to Q-learning results in Self-correcting Q-learning. We show theoretically that this new algorithm enjoys the same convergence guarantees as Q-learning while being more accurate. Empirically, it performs better than Double Q-learning in domains with rewards of high variance, and it even attains faster convergence than Q-learning in domains with rewards of zero or low variance. These advantages transfer to a Deep Q Network implementation that we call Self-correcting DQN and which outperforms regular DQN and Double DQN on several tasks in the Atari 2600 domain.