mlsanityData to 7 Oct 2026

Dropout Q-Functions for Doubly Efficient Reinforcement Learning

arXiv 2110.02034 ↗First posted 5 Oct 2021cs.LG cs.AIcited by 108

Not on the map. Why?

Abstract (arXiv)

Randomized ensembled double Q-learning (REDQ) (Chen et al., 2021b) has recently achieved state-of-the-art sample efficiency on continuous-action reinforcement learning benchmarks. This superior sample efficiency is made possible by using a large Q-function ensemble. However, REDQ is much less computationally efficient than non-ensemble counterparts such as Soft Actor-Critic (SAC) (Haarnoja et al., 2018a). To make REDQ more computationally efficient, we propose a method of improving computational efficiency called DroQ, which is a variant of REDQ that uses a small ensemble of dropout Q-functions. Our dropout Q-functions are simple Q-functions equipped with dropout connection and layer normalization. Despite its simplicity of implementation, our experimental results indicate that DroQ is doubly (sample and computationally) efficient. It achieved comparable sample efficiency with REDQ, much better computational efficiency than REDQ, and comparable computational efficiency with that of SAC.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2016Layer Normalizationcited by 5,579
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2018Group Normalizationcited by 1,514
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
29 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  8. 2018Maximum a Posteriori Policy Optimisationcited by 318
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  10. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  11. 2017Concrete Dropoutcited by 227
  12. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  13. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  14. 2019Understanding and Improving Layer Normalizationcited by 213
  15. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  16. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  17. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  18. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  19. 2020Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learningcited by 95
  20. 2020Masksembles for Uncertainty Estimationcited by 88
  21. 2017Uncertainty-Aware Reinforcement Learning for Collision Avoidancecited by 86
  22. 2020Bidirectional Model-based Policy Optimizationcited by 38
  23. 2019Guided Meta-Policy Searchcited by 26
  24. 2020Model-based Policy Optimization with Unsupervised Model Adaptationcited by 16
  25. 2017Efficient exploration with Double Uncertain Value Networkscited by 15
  26. 2021Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddingscited by 12
  27. 2020Meta-Model-Based Meta-Policy Optimizationcited by 7
  28. 2020Model-Based Generalization Under Parameter Uncertainty Using Path Integral Controlcited by 7
  29. 2021MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 24; 2024: 31; 2025: 26; 2026: 20.

108 papers.

Sort:

By area:

  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  4. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  5. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  6. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  7. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  8. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  9. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  10. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  11. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learningcited by 50
  12. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
38 more
  1. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  2. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  3. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  4. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  5. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  6. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  7. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30
  8. 2023RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learningcited by 29
  9. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
  10. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 23
  11. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
  12. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  13. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 20
  14. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  15. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergencecited by 17
  16. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learningcited by 17
  17. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  18. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 15
  19. 2024Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Modelscited by 14
  20. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  21. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  22. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  23. 2023Mind the Model, Not the Agent: The Primacy Bias in Model-based RLcited by 9
  24. 2023RL-X: A Deep Reinforcement Learning Library (not only) for RoboCupcited by 7
  25. 2023Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deploymentcited by 7
  26. 2024Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPscited by 7
  27. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learningcited by 7
  28. 2025Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotioncited by 7
  29. 2023On the Theory of Risk-Aware Agents: Bridging Actor-Critic and Economicscited by 6
  30. 2024Iterated Q-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learningcited by 6
  31. 2024PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learningcited by 6
  32. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  33. 2023Understanding the Synergies between Quality-Diversity and Deep Reinforcement Learningcited by 5
  34. 2024Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learningcited by 5
  35. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  36. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  37. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  38. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4

50 most cited of 108 All 108 (JSON)