mlsanityData to 7 Oct 2026

Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Tasks

arXiv 2203.11526 ↗First posted 22 Mar 2022cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Double Q-learning is a popular reinforcement learning algorithm in Markov decision process (MDP) problems. Clipped Double Q-learning, as an effective variant of Double Q-learning, employs the clipped double estimator to approximate the maximum expected action value. Due to the underestimation bias of the clipped double estimator, the performance of clipped Double Q-learning may be degraded in some stochastic environments. In this paper, in order to reduce the underestimation bias, we propose an action candidate-based clipped double estimator for Double Q-learning. Specifically, we first select a set of elite action candidates with high action values from one set of estimators. Then, among these candidates, we choose the highest valued action from the other set of estimators. Finally, we use the maximum value in the second set of estimators to clip the action value of the chosen action in the first set of estimators and the clipped value is used for approximating the maximum expected action value. Theoretically, the underestimation bias in our clipped Double Q-learning decays monotonically as the number of action candidates decreases. Moreover, the number of action candidates controls the trade-off between the overestimation and underestimation biases. In addition, we also extend our clipped Double Q-learning to continuous action tasks via approximating the elite continuous action candidates. We empirically verify that our algorithm can more accurately estimate the maximum expected action value on some toy environments and yield good performance on several benchmark problems.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 19 of its 47 references that are in the map. First posted on arXiv 22 Mar 2022.

Builds on

19 of 47 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  9. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  10. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  11. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  12. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
7 more
  1. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  2. 2015Deep Attention Recurrent Q-Networkcited by 27
  3. 2019MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experimentscited by 17
  4. 2020Finite-Time Analysis for Double Q-learningcited by 12
  5. 2013Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Averagecited by 7
  6. 2021Action Candidate Based Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 7
  7. 2020Deep Reinforcement Learning with Weighted Q-Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 6.

8 papers.

  1. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  2. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  3. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  4. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Modelscited by 7
  5. 2025FUSER: Feed-Forward MUltiview 3D Registration Transformer and SE(3)N Diffusion Refinementcited by 3
  6. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
  7. 2025Frictional Q-Learningcited by 0
  8. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularizationcited by 0