mlsanityData to 7 Oct 2026

Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

arXiv 2101.05982 ↗First posted 15 Jan 2021cs.LG cs.AIcited by 278

Not on the map. Why?

Cited by 35 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio >> 1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio >> 1.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 15 Jan 2021.

Builds on

30 of 50 references.

References · 30

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
18 more
  1. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  2. 2017Noisy Networks for Explorationcited by 262
  3. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  4. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  5. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  6. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  7. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  8. 2019When to use parametric models in reinforcement learning?cited by 118
  9. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  10. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  11. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  12. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  13. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  14. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  15. 2020Bidirectional Model-based Policy Optimizationcited by 38
  16. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  17. 2020The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learningcited by 14
  18. 2020Maximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errorscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 32; 2023: 50; 2024: 74; 2025: 63; 2026: 47.

278 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  4. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  5. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  6. 2023Synthetic Experience Replaycited by 100
  7. 2022Accelerated Policy Learning with Parallel Differentiable Simulationcited by 92
  8. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  9. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  10. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  11. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  12. 2024Simplifying Deep Temporal Difference Learningcited by 71
38 more
  1. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  2. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  3. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  4. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  5. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  6. 2022Offline RL Policies Should be Trained to be Adaptivecited by 52
  7. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  8. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  9. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  10. 2022Optimizing Sequential Experimental Design with Deep Reinforcement Learningcited by 45
  11. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  12. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 43
  13. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  14. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  15. 2023Finetuning Offline World Models in the Real Worldcited by 39
  16. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  17. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  18. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  19. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  20. 2022Hierarchical Reinforcement Learning for Precise Soccer Shooting Skills using a Quadrupedal Robotcited by 32
  21. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  22. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  23. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  24. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  25. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  26. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  27. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  28. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  29. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  30. 2021Sample-Efficient Reinforcement Learning via Conservative Model-Based Actor-Criticcited by 23
  31. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  32. 2024Adaptive teachers for amortized samplerscited by 23
  33. 2021Generalizable Episodic Memory for Deep Reinforcement Learningcited by 22
  34. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
  35. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  36. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  37. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  38. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20

50 most cited of 278 All 278 (JSON)