mlsanityData to 13 Oct 2026

The Curse of Diversity in Ensemble-Based Exploration

arXiv 2405.04342 ↗First posted 7 May 2024cs.LGcited by 6

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We uncover a surprising phenomenon in deep reinforcement learning: training a diverse ensemble of data-sharing agents -- a well-established exploration strategy -- can significantly impair the performance of the individual ensemble members when compared to standard single-agent training. Through careful analysis, we attribute the degradation in performance to the low proportion of self-generated data in the shared training data for each ensemble member, as well as the inefficiency of the individual ensemble members to learn from such highly off-policy data. We thus name this phenomenon the curse of diversity. We find that several intuitive solutions -- such as a larger replay buffer or a smaller ensemble size -- either fail to consistently mitigate the performance loss or undermine the advantages of ensembling. Finally, we demonstrate the potential of representation learning to counteract the curse of diversity with a novel method named Cross-Ensemble Representation Learning (CERL) in both discrete and continuous control domains. Our work offers valuable insights into an unexpected pitfall in ensemble-based exploration and raises important caveats for future applications of similar approaches.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 56 of its 68 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

56 of 68 references.

References · 56

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,710
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,231
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,185
  6. 2015Prioritized Experience Replaycited by 1,078
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 871
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 754
  11. 2017Deep Mutual Learningcited by 624
  12. 2016Deep Exploration via Bootstrapped DQNcited by 520
38 more
  1. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 465
  2. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 403
  3. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 311
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 298
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 279
  6. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 268
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 243
  8. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  9. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 216
  10. 2016Learning to Navigate in Complex Environmentscited by 193
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 173
  12. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  13. 2018Large scale distributed neural network training through online distillationcited by 145
  14. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  15. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  16. 2016Opponent Modeling in Deep Reinforcement Learningcited by 95
  17. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  18. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  19. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 58
  20. 2020Peer Collaborative Learning for Online Knowledge Distillationcited by 58
  21. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  22. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  23. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  24. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  25. 2023Efficient Deep Reinforcement Learning Requires Regulating Overfittingcited by 36
  26. 2021Ensemble Bootstrapping for Q-Learningcited by 31
  27. 2022The Phenomenon of Policy Churncited by 28
  28. 2018Mix&Match - Agent Curricula for Reinforcement Learningcited by 26
  29. 2017A Deep Policy Inference Q-Network for Multi-Agent Systemscited by 24
  30. 2019Agent Modeling as Auxiliary Task for Deep Reinforcement Learningcited by 14
  31. 2020Non-local Policy Optimization via Diversity-regularized Collaborative Explorationcited by 13
  32. 2022Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networkscited by 13
  33. 2019Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learningcited by 11
  34. 2020Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillationcited by 10
  35. 2021Continuous Control With Ensemble Deep Deterministic Policy Gradientscited by 8
  36. 2021Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performancecited by 7
  37. 2022Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shapingcited by 6
  38. 2022Improving the Diversity of Bootstrapped DQN by Replacing Priors With Noisecited by 3

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  2. 2025A Novel Framework for Uncertainty-Driven Adaptive Explorationcited by 1
  3. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  4. 2026The Essence of Balance for Self-Improving Agents in Vision-and-Language Navigationcited by 1
  5. 2025An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learningcited by 0
  6. 2025Value of Information-Enhanced Exploration in Bootstrapped DQNcited by 0