mlsanityData to 7 Oct 2026

Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance

arXiv 2111.09159 ↗First posted 17 Nov 2021cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent advances in model-free deep reinforcement learning (DRL) show that simple model-free methods can be highly effective in challenging high-dimensional continuous control tasks. In particular, Truncated Quantile Critics (TQC) achieves state-of-the-art asymptotic training performance on the MuJoCo benchmark with a distributional representation of critics; and Randomized Ensemble Double Q-Learning (REDQ) achieves high sample efficiency that is competitive with state-of-the-art model-based methods using a high update-to-data ratio and target randomization. In this paper, we propose a novel model-free algorithm, Aggressive Q-Learning with Ensembles (AQE), which improves the sample-efficiency performance of REDQ and the asymptotic performance of TQC, thereby providing overall state-of-the-art performance during all stages of training. Moreover, AQE is very simple, requiring neither distributional representation of critics nor target randomization. The effectiveness of AQE is further supported by our extensive experiments, ablations, and theoretical results.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 19 of its 32 references that are in the map. First posted on arXiv 17 Nov 2021.

Builds on

19 of 32 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2017Deep Reinforcement Learning that Matterscited by 727
  9. 2016Deep Exploration via Bootstrapped DQNcited by 518
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
7 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  2. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  3. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  4. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 138
  5. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  6. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  7. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 3.

7 papers.

  1. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  2. 2023RL-X: A Deep Reinforcement Learning Library (not only) for RoboCupcited by 7
  3. 2025Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotioncited by 7
  4. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  5. 2024AFU: Actor-Free critic Updates in off-policy RL for continuous controlcited by 1
  6. 2025Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimizationcited by 0
  7. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0