mlsanityData to 14 Oct 2026

AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization

arXiv 2506.05634 ↗First posted 5 Jun 2025cs.LG cs.AI cs.NEcited by 3

Not on the map. Why?

Abstract (arXiv)

Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Leveraging the equivalence between policies and occupancy measures, we present a theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Maximum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low-dimensional projection of these embeddings that captures the most behaviorally significant dimensions can then be used as behavioral descriptors for CMA-MAE, a state of the art blackbox QD method, to discover diverse policies. We prove that our embeddings converge to true MMD distances between occupancy measures as the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks we demonstrate AutoQD's ability in discovering diverse policies without predefined behavioral descriptors, presenting a well-motivated alternative to prior methods in unsupervised Reinforcement Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specific knowledge. Source code is available at https://github.com/conflictednerd/autoqd-code.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

32 of 50 references.

References · 32

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  3. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  5. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  6. 2014Robots that can adapt like animalscited by 204
  7. 2016Generalization Properties of Learning with Random Featurescited by 164
  8. 2020Effective Diversity in Population Based Reinforcement Learningcited by 119
  9. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  10. 2019Covariance Matrix Adaptation for the Rapid Illumination of Behavior Spacecited by 78
  11. 2021Differentiable Quality Diversitycited by 72
  12. 2019Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptorscited by 57
20 more
  1. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 54
  2. 2020Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Networkcited by 52
  3. 2018Structured Evolution with Compact Architectures for Scalable Policy Optimizationcited by 48
  4. 2019Procedural Content Generation through Quality Diversitycited by 45
  5. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  6. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 38
  7. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  8. 2019Learning to Score Behaviors for Guided Policy Optimizationcited by 34
  9. 2021Unsupervised Behaviour Discovery with Quality-Diversity Optimisationcited by 30
  10. 2023pyribs: A Bare-Bones Python Library for Quality Diversity Optimizationcited by 30
  11. 2022Covariance Matrix Adaptation MAP-Annealingcited by 25
  12. 2019Unsupervised Learning and Exploration of Reachable Outcome Spacecited by 23
  13. 2022Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discoverycited by 23
  14. 2023Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learningcited by 20
  15. 2024Quality-Weighted Vendi Scores And Their Application To Diverse Experimental Designcited by 17
  16. 2019Efficacy of Modern Neuro-Evolutionary Strategies for Continuous Control Optimizationcited by 15
  17. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 11
  18. 2022Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealingcited by 10
  19. 2023Gradient-Informed Quality Diversity for the Illumination of Discrete Spacescited by 4
  20. 2022Discovering Unsupervised Behaviours from Full-State Trajectoriescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Soft Quality-Diversity Optimizationcited by 1
  2. 2026Diverse and Adaptable Arm Coordination for Octopus-Crawling via Diffusion-Based Uncertainty-Aware Optimizationcited by 1
  3. 2026Quality-Diversity Optimization as Multi-Objective Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vector Quantized-Elites: Unsupervised and Problem-Agnostic Quality-Diversity Optimization
  2. 2026Behavioral Foundation Models for Quality Diversity
  3. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Models
  4. 2023Diversity from Human Feedback
  5. 2024Measuring Policy Distance for Multi-Agent Reinforcement Learning
  6. 2023Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization
  7. 2024Quality Diversity for Robot Learning: Limitations and Future Directions
  8. 2025State Diversity Matters in Offline Behavior Distillation