# Synergizing Quality-Diversity with Descriptor-Conditioned Reinforcement Learning

arXiv: https://arxiv.org/abs/2401.08632 · HTML: https://mlsanity.com/paper/2401.08632/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 53 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Builds on (most cited first, 30 of 39)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [Differentiable Quality Diversity](https://mlsanity.com/paper/2106.03894/) (2021)
- [Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithm](https://mlsanity.com/paper/1610.05729/) (2016)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization](https://mlsanity.com/paper/2006.08505/) (2020)
- [The Difficulty of Passive Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2110.14020/) (2021)
- [Discovering the Elite Hypervolume by Leveraging Interspecies Correlation](https://mlsanity.com/paper/1804.03906/) (2018)
- [Scaling MAP-Elites to Deep Neuroevolution](https://mlsanity.com/paper/2003.01825/) (2020)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2205.10752/) (2022)
- [Reset-free Trial-and-Error Learning for Robot Damage Recovery](https://mlsanity.com/paper/1610.04213/) (2016)

## Built on by (most cited first, 6 of 6)

- [Dominated Novelty Search: Rethinking Local Competition in Quality-Diversity](https://mlsanity.com/paper/2502.00593/) (2025)
- [Preference-Conditioned Gradient Variations for Multi-Objective Quality-Diversity](https://mlsanity.com/paper/2411.12433/) (2024)
- [Diverse Approaches to Optimal Execution Schedule Generation](https://mlsanity.com/paper/2601.22113/) (2026)
- [Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning](https://mlsanity.com/paper/2606.08735/) (2026)
- [GAE: Graph-Augmented Evolution for Scientific Discovery via Reinforcement Optimization](https://mlsanity.com/paper/2607.10127/) (2026)
- [Behavioral Foundation Models for Quality Diversity](https://mlsanity.com/paper/2609.35615/) (2026)

## Abstract (arXiv)

A hallmark of intelligence is the ability to exhibit a wide range of effective behaviors. Inspired by this principle, Quality-Diversity algorithms, such as MAP-Elites, are evolutionary methods designed to generate a set of diverse and high-fitness solutions. However, as a genetic algorithm, MAP-Elites relies on random mutations, which can become inefficient in high-dimensional search spaces, thus limiting its scalability to more complex domains, such as learning to control agents directly from high-dimensional inputs. To address this limitation, advanced methods like PGA-MAP-Elites and DCG-MAP-Elites have been developed, which combine actor-critic techniques from Reinforcement Learning with MAP-Elites, significantly enhancing the performance and efficiency of Quality-Diversity algorithms in complex, high-dimensional tasks. While these methods have successfully leveraged the trained critic to guide more effective mutations, the potential of the trained actor remains underutilized in improving both the quality and diversity of the evolved population. In this work, we introduce DCRL-MAP-Elites, an extension of DCG-MAP-Elites that utilizes the descriptor-conditioned actor as a generative model to produce diverse solutions, which are then injected into the offspring batch at each generation. Additionally, we present an empirical analysis of the fitness and descriptor reproducibility of the solutions discovered by each algorithm. Finally, we present a second empirical analysis shedding light on the synergies between the different variations operators and explaining the performance improvement from PGA-MAP-Elites to DCRL-MAP-Elites.