mlsanityData to 14 Oct 2026

Effective Diversity in Population Based Reinforcement Learning

arXiv 2002.00632 ↗First posted 3 Feb 2020cs.LG stat.MLcited by 119

Not on the map. Why?

Abstract (arXiv)

Exploration is a key problem in reinforcement learning, since agents can only learn from data they acquire in the environment. With that in mind, maintaining a population of agents is an attractive method, as it allows data be collected with a diverse set of behaviors. This behavioral diversity is often boosted via multi-objective loss functions. However, those approaches typically leverage mean field updates based on pairwise distances, which makes them susceptible to cycling behaviors and increased redundancy. In addition, explicitly boosting diversity often has a detrimental impact on optimizing already fruitful behaviors for rewards. As such, the reward-diversity trade off typically relies on heuristics. Finally, such methods require behavioral representations, often handcrafted and domain specific. In this paper, we introduce an approach to optimize all members of a population simultaneously. Rather than using pairwise distance, we measure the volume of the entire population in a behavioral manifold, defined by task-agnostic behavioral embeddings. In addition, our algorithm Diversity via Determinants (DvD), adapts the degree of diversity during training using online learning techniques. We introduce both evolutionary and gradient-based instantiations of DvD and show they effectively improve exploration without reducing performance when better exploration is not required.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 31; 2023: 30; 2024: 6; 2025: 18; 2026: 10.

119 papers.

Sort:

By area:

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 106
  3. 2021Differentiable Quality Diversitycited by 72
  4. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  5. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  6. 2022Winner Takes It All: Training Performant RL Populations for Combinatorial Optimizationcited by 58
  7. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  8. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 41
  9. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  10. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 38
  11. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  12. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
38 more
  1. 2023Evolutionary Reinforcement Learning: A Surveycited by 29
  2. 2021Active Offline Policy Selectioncited by 26
  3. 2024Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithmscited by 26
  4. 2022Covariance Matrix Adaptation MAP-Annealingcited by 25
  5. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  6. 2021Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systemscited by 24
  7. 2022Learning High-DOF Reaching-and-Grasping via Dynamic Representation of Gripper-Object Interactioncited by 24
  8. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 24
  9. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 23
  10. 2022Continuously Discovering Novel Strategies via Reward-Switching Policy Optimizationcited by 23
  11. 2022Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discoverycited by 23
  12. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  13. 2022Bayesian Generational Population-Based Trainingcited by 20
  14. 2023TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Playcited by 20
  15. 2021Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Informationcited by 19
  16. 2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
  17. 2021Understanding the Effect of Stochasticity in Policy Optimizationcited by 18
  18. 2022Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemblecited by 18
  19. 2022Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversitycited by 17
  20. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Gamescited by 17
  21. 2023Policy Space Diversity for Non-Transitive Gamescited by 15
  22. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  23. 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
  24. 2022ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representationcited by 14
  25. 2023System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learningcited by 14
  26. 2023Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selectioncited by 14
  27. 2021Sparse Reward Exploration via Novelty Search and Emitterscited by 13
  28. 2021Adaptable Agent Populations via a Generative Model of Policiescited by 12
  29. 2022Heterogeneous Multi-agent Zero-Shot Coordination by Coevolutioncited by 12
  30. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  31. 2022DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimizationcited by 10
  32. 2022Learning Robust Real-Time Cultural Transmission without Human Datacited by 9
  33. 2022Hierarchically Structured Task-Agnostic Continual Learningcited by 9
  34. 2023Generalization through Diversity: Improving Unsupervised Environment Designcited by 9
  35. 2021Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRLcited by 8
  36. 2021Mixture-of-Variational-Experts for Continual Learningcited by 8
  37. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  38. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8

50 most cited of 119 All 119 (JSON)