# Efficient Quality-Diversity Optimization through Diverse Quality Species

arXiv: https://arxiv.org/abs/2304.07425 · HTML: https://mlsanity.com/paper/2304.07425/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 14 Apr 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Equivalence Between Policy Gradients and Soft Q-Learning](https://mlsanity.com/paper/1704.06440/) (2017)
- [Planning with Goal-Conditioned Policies](https://mlsanity.com/paper/1911.08453/) (2019)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space](https://mlsanity.com/paper/1912.02400/) (2019)
- [Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithm](https://mlsanity.com/paper/1610.05729/) (2016)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization](https://mlsanity.com/paper/2006.08505/) (2020)
- [Procedural Content Generation through Quality Diversity](https://mlsanity.com/paper/1907.04053/) (2019)
- [Discovering the Elite Hypervolume by Leveraging Interspecies Correlation](https://mlsanity.com/paper/1804.03906/) (2018)
- [Scaling MAP-Elites to Deep Neuroevolution](https://mlsanity.com/paper/2003.01825/) (2020)
- [Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuilding](https://mlsanity.com/paper/2112.03534/) (2021)
- [Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires](https://mlsanity.com/paper/2109.08522/) (2021)
- [Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains](https://mlsanity.com/paper/2210.13156/) (2022)
- [Exploring the Evolution of GANs through Quality Diversity](https://mlsanity.com/paper/2007.06251/) (2020)
- [Map-based Multi-Policy Reinforcement Learning: Enhancing Adaptability of Robots by Deep Reinforcement Learning](https://mlsanity.com/paper/1710.06117/) (2017)

## Built on by (most cited first, 3 of 3)

- [Can ChatGPT Enable ITS? The Case of Mixed Traffic Control via Reinforcement Learning](https://mlsanity.com/paper/2306.08094/) (2023)
- [Agent Skill Acquisition for Large Language Models via CycleQD](https://mlsanity.com/paper/2410.14735/) (2024)
- [Analyzing Behaviors of Mixed Traffic via Reinforcement Learning at Unsignalized Intersections](https://mlsanity.com/paper/2312.05325/) (2023)

## Abstract (arXiv)

A prevalent limitation of optimizing over a single objective is that it can be misguided, becoming trapped in local optimum. This can be rectified by Quality-Diversity (QD) algorithms, where a population of high-quality and diverse solutions to a problem is preferred. Most conventional QD approaches, for example, MAP-Elites, explicitly manage a behavioral archive where solutions are broken down into predefined niches. In this work, we show that a diverse population of solutions can be found without the limitation of needing an archive or defining the range of behaviors in advance. Instead, we break down solutions into independently evolving species and use unsupervised skill discovery to learn diverse, high-performing solutions. We show that this can be done through gradient-based mutations that take on an information theoretic perspective of jointly maximizing mutual information and performance. We propose Diverse Quality Species (DQS) as an alternative to archive-based QD algorithms. We evaluate it over several simulated robotic environments and show that it can learn a diverse set of solutions from varying species. Furthermore, our results show that DQS is more sample-efficient and performant when compared to other QD algorithms. Relevant code and hyper-parameters are available at: https://github.com/rwickman/NEAT_RL.