Ensemble Elastic DQN: A Step Dependent Ensemble Approach for Reducing Overestimation in Deep Value-Based Reinforcement Learning
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Deep Q-Networks (DQN) can suffer from overestimation bias because bootstrapped targets use a maximisation operation over noisy value estimates. Ensemble-based methods and multi-step methods have each been used to improve the stability and sample efficiency of value-based reinforcement learning, but their interaction remains less well understood. This paper introduces Ensemble Elastic DQN (EEDQN), a value-based reinforcement learning algorithm that combines adaptive elastic multi-step returns with ensemble-based target aggregation. EEDQN replaces the clustering-based state similarity test used in earlier Elastic Step DQN with a lightweight Q-value difference rule, making adaptive return construction simpler to apply in discrete control settings. The method then applies horizon dependent ensemble aggregation, one-step targets use the ensemble mean, while longer elastic returns use the ensemble minimum. This design aims to reduce extreme optimistic bootstrap estimates without making every update uniformly conservative. We evaluate EEDQN on five MinAtar environments against DQN, Double DQN, Averaged DQN, MaxMin DQN, and Elastic Step DQN. EEDQN achieves the highest final return in four of the five environments and remains competitive with conservative ensemble baselines. An aggregation rule ablation shows that the best degree of conservatism is environment dependent, suggesting that adaptive return length and ensemble aggregation interact in non-trivial ways.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 18 references that are in the map. First posted on arXiv 6 Jun 2025.
Builds on
13 of 18 references.
References · 13
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
- 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
- 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
- 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
- 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
- 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
- 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
- 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
- 2020WD3: Taming the Estimation Bias in Deep Reinforcement Learningcited by 16
- 2022Elastic Step DQN: A novel multi-step algorithm to alleviate overestimation in Deep QNetworkscited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025GB-DQN: Gradient Boosted DQN Models for Non-stationary Reinforcement Learning
- 2025Dual Ensembled Multiagent Q-Learning with Hypernet Regularizer
- 2025Enhancing Q-Value Updates in Deep Q-Learning via Successor-State Prediction
- 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learning
- 2025β-DQN: Improving Deep Q-Learning By Evolving the Behavior
- 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values
- 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
- 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes