# WD3: Taming the Estimation Bias in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2006.12622 · HTML: https://mlsanity.com/paper/2006.12622/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 16 of 16)

- [Order Matters: Agent-by-agent Policy Optimization](https://mlsanity.com/paper/2302.06205/) (2023)
- [Frustratingly Easy Regularization on Representation Can Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2205.14557/) (2022)
- [Adaptive Regularization of Representation Rank as an Implicit Constraint of Bellman Equation](https://mlsanity.com/paper/2404.12754/) (2024)
- [MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2109.10552/) (2021)
- [Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworks](https://mlsanity.com/paper/2210.03325/) (2022)
- [Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control](https://mlsanity.com/paper/2310.11138/) (2023)
- [Optimistic critics can empower small actors](https://mlsanity.com/paper/2506.01016/) (2025)
- [Estimation Error Correction in Deep Reinforcement Learning for Deterministic Actor-Critic Methods](https://mlsanity.com/paper/2109.10736/) (2021)
- [Automating Control of Overestimation Bias for Reinforcement Learning](https://mlsanity.com/paper/2110.13523/) (2021)
- [Eigensubspace of Temporal-Difference Dynamics and How It Improves Value Approximation in Reinforcement Learning](https://mlsanity.com/paper/2306.16750/) (2023)
- [Deep Reinforcement Learning in Applied Control: Challenges, Analysis, and Insights](https://mlsanity.com/paper/2507.08196/) (2025)
- [AWD3: Dynamic Reduction of the Estimation Bias](https://mlsanity.com/paper/2111.06780/) (2021)
- [Centralized Cooperative Exploration Policy for Continuous Control Tasks](https://mlsanity.com/paper/2301.02375/) (2023)
- [Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradients](https://mlsanity.com/paper/2109.11788/) (2021)
- [Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning](https://mlsanity.com/paper/2506.05716/) (2025)
- [ProSh: Probabilistic Shielding for Model-free Reinforcement Learning](https://mlsanity.com/paper/2510.15720/) (2025)

## Abstract (arXiv)

The overestimation phenomenon caused by function approximation is a well-known issue in value-based reinforcement learning algorithms such as deep Q-networks and DDPG, which could lead to suboptimal policies. To address this issue, TD3 takes the minimum value between a pair of critics. In this paper, we show that the TD3 algorithm introduces underestimation bias in mild assumptions. To obtain a more precise estimation for value function, we unify these two opposites and propose a novel algorithm underlineWeighted underlineDelayed underlineDeep underlineDeterministic Policy Gradient (WD3), which can eliminate the estimation bias and further improve the performance by weighting a pair of critics. To demonstrate the effectiveness of WD3, we compare the learning process of value function between DDPG, TD3, and WD3. The results verify that our algorithm does eliminate the estimation error of value functions. Furthermore, we evaluate our algorithm on the continuous control tasks. We observe that in each test task, the performance of WD3 consistently outperforms, or at the very least matches, that of the state-of-the-art algorithmsfootnoteOur code is available at~hrefhttps://sites.google.com/view/ictai20-wd3/https://sites.google.com/view/ictai20-wd3/..