# AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization

arXiv: https://arxiv.org/abs/2506.05634 · HTML: https://mlsanity.com/paper/2506.05634/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 5 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space](https://mlsanity.com/paper/1912.02400/) (2019)
- [Differentiable Quality Diversity](https://mlsanity.com/paper/2106.03894/) (2021)
- [Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors](https://mlsanity.com/paper/1905.11874/) (2019)
- [Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization](https://mlsanity.com/paper/2006.08505/) (2020)
- [Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Network](https://mlsanity.com/paper/2007.05674/) (2020)
- [Structured Evolution with Compact Architectures for Scalable Policy Optimization](https://mlsanity.com/paper/1804.02395/) (2018)
- [Procedural Content Generation through Quality Diversity](https://mlsanity.com/paper/1907.04053/) (2019)
- [Deep Surrogate Assisted Generation of Environments](https://mlsanity.com/paper/2206.04199/) (2022)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Learning to Score Behaviors for Guided Policy Optimization](https://mlsanity.com/paper/1906.04349/) (2019)
- [Unsupervised Behaviour Discovery with Quality-Diversity Optimisation](https://mlsanity.com/paper/2106.05648/) (2021)
- [pyribs: A Bare-Bones Python Library for Quality Diversity Optimization](https://mlsanity.com/paper/2303.00191/) (2023)
- [Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2205.10752/) (2022)
- [Unsupervised Learning and Exploration of Reachable Outcome Space](https://mlsanity.com/paper/1909.05508/) (2019)
- [Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery](https://mlsanity.com/paper/2210.03516/) (2022)
- [Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning](https://mlsanity.com/paper/2305.13795/) (2023)
- [Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Design](https://mlsanity.com/paper/2405.02449/) (2024)
- [Efficacy of Modern Neuro-Evolutionary Strategies for Continuous Control Optimization](https://mlsanity.com/paper/1912.05239/) (2019)
- [Self-Supervised Reinforcement Learning that Transfers using Random Features](https://mlsanity.com/paper/2305.17250/) (2023)
- [Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2210.02622/) (2022)

## Built on by (most cited first, 3 of 3)

- [Soft Quality-Diversity Optimization](https://mlsanity.com/paper/2512.00810/) (2025)
- [Diverse and Adaptable Arm Coordination for Octopus-Crawling via Diffusion-Based Uncertainty-Aware Optimization](https://mlsanity.com/paper/2609.21138/) (2026)
- [Quality-Diversity Optimization as Multi-Objective Optimization](https://mlsanity.com/paper/2602.00478/) (2026)

## Abstract (arXiv)

Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Leveraging the equivalence between policies and occupancy measures, we present a theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Maximum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low-dimensional projection of these embeddings that captures the most behaviorally significant dimensions can then be used as behavioral descriptors for CMA-MAE, a state of the art blackbox QD method, to discover diverse policies. We prove that our embeddings converge to true MMD distances between occupancy measures as the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks we demonstrate AutoQD's ability in discovering diverse policies without predefined behavioral descriptors, presenting a well-motivated alternative to prior methods in unsupervised Reinforcement Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specific knowledge. Source code is available at https://github.com/conflictednerd/autoqd-code.