ADDQ: Adaptive Distributional Double Q-Learning
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Bias problems in the estimation of Q-values are a well-known obstacle that slows down convergence of Q-learning and actor-critic methods. One of the reasons of the success of modern RL algorithms is partially a direct or indirect overestimation reduction mechanism. We propose an easy to implement method built on top of distributional reinforcement learning (DRL) algorithms to deal with the overestimation in a locally adaptive way. Our framework is simple to implement, existing distributional algorithms can be improved with a few lines of code. We provide theoretical evidence and use double Q-learning to show how to include locally adaptive overestimation control in existing algorithms. Experiments are provided for tabular, Atari, and MuJoCo environments.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 44 references that are in the map. First posted on arXiv 24 Jun 2025.
Builds on
21 of 44 references.
References · 21
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
- 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
- 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 126
- 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
- 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
- 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
9 more
- 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
- 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
- 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
- 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
- 2021Ensemble Bootstrapping for Q-Learningcited by 31
- 2023An Analysis of Quantile Temporal-Difference Learningcited by 31
- 2023The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimationcited by 14
- 2021Adaptively Calibrated Critic Estimates for Deep Reinforcement Learningcited by 6
- 2018The Potential of the Return Distribution for Exploration in RLcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Distributional Active Inferencecited by 0
- 2026Distributional Active Inference28 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2024Finite-Time Analysis of Simultaneous Double Q-learning
- 2025Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss
- 2024Exploiting Estimation Bias in Clipped Double Q-Learning for Continous Control Reinforcement Learning Tasks
- 2025Deep Double Q-learning
- 2024Double Successive Over-Relaxation Q-Learning with an Extension to Deep Reinforcement Learning
- 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
- 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
- 2025Dual Ensembled Multiagent Q-Learning with Hypernet Regularizer