mlsanityData to 14 Oct 2026

Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks

arXiv 2402.09078 ↗First posted 14 Feb 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Continuous control Deep Reinforcement Learning (RL) approaches are known to suffer from estimation biases, leading to suboptimal policies. This paper introduces innovative methods in RL, focusing on addressing and exploiting estimation biases in Actor-Critic methods for continuous control tasks, using Deep Double Q-Learning. We design a Bias Exploiting (BE) mechanism to dynamically select the most advantageous estimation bias during training of the RL agent. Most State-of-the-art Deep RL algorithms can be equipped with the BE mechanism, without hindering performance or computational complexity. Our extensive experiments across various continuous control tasks demonstrate the effectiveness of our approaches. We show that RL algorithms equipped with this method can match or surpass their counterparts, particularly in environments where estimation biases significantly impact learning. The results underline the importance of bias exploitation in improving policy learning in RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 8 of its 18 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

8 of 18 references.

References · 8

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  6. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  7. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  8. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Spacecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization
  2. 2025ADDQ: Adaptive Distributional Double Q-Learning
  3. 2025Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss
  4. 2025Deep Double Q-learning
  5. 2021Q-Value Weighted Regression: Reinforcement Learning with Limited Data
  6. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  7. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  8. 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functions