# Don't Bet on Luck Alone: Enhancing Behavioral Reproducibility of Quality-Diversity Solutions in Uncertain Domains

arXiv: https://arxiv.org/abs/2304.03672 · HTML: https://mlsanity.com/paper/2304.03672/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 24 of 24)

- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space](https://mlsanity.com/paper/1912.02400/) (2019)
- [Differentiable Quality Diversity](https://mlsanity.com/paper/2106.03894/) (2021)
- [Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Network](https://mlsanity.com/paper/2007.05674/) (2020)
- [Scaling MAP-Elites to Deep Neuroevolution](https://mlsanity.com/paper/2003.01825/) (2020)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Illuminating Diverse Neural Cellular Automata for Level Generation](https://mlsanity.com/paper/2109.05489/) (2021)
- [Reset-free Trial-and-Error Learning for Robot Damage Recovery](https://mlsanity.com/paper/1610.04213/) (2016)
- [Multi-Objective Quality Diversity Optimization](https://mlsanity.com/paper/2202.03057/) (2022)
- [Empirical analysis of PGA-MAP-Elites for Neuroevolution in Uncertain Domains](https://mlsanity.com/paper/2210.13156/) (2022)
- [Accelerated Quality-Diversity through Massive Parallelism](https://mlsanity.com/paper/2202.01258/) (2022)
- [Fast and stable MAP-Elites in noisy domains using deep grids](https://mlsanity.com/paper/2006.14253/) (2020)
- [Uncertain Quality-Diversity: Evaluation methodology and new methods for Quality-Diversity in Uncertain Domains](https://mlsanity.com/paper/2302.00463/) (2023)
- [Hierarchical Behavioral Repertoires with Unsupervised Descriptors](https://mlsanity.com/paper/1804.07127/) (2018)
- [Adaptive Prior Selection for Repertoire-based Online Adaptation in Robotics](https://mlsanity.com/paper/1907.07029/) (2019)
- [Benchmarking Quality-Diversity Algorithms on Neuroevolution for Reinforcement Learning](https://mlsanity.com/paper/2211.02193/) (2022)
- [ARCH-Elites: Quality-Diversity for Urban Design](https://mlsanity.com/paper/2104.08774/) (2021)
- [Hierarchical Quality-Diversity for Online Damage Recovery](https://mlsanity.com/paper/2204.05726/) (2022)

## Built on by (most cited first, 3 of 3)

- [Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics](https://mlsanity.com/paper/2403.09930/) (2024)
- [Quality-Diversity with Limited Resources](https://mlsanity.com/paper/2406.03731/) (2024)
- [Quality Diversity for Reliable Data Driven Time-Use Optimization](https://mlsanity.com/paper/2608.05230/) (2026)

## Abstract (arXiv)

Quality-Diversity (QD) algorithms are designed to generate collections of high-performing solutions while maximizing their diversity in a given descriptor space. However, in the presence of unpredictable noise, the fitness and descriptor of the same solution can differ significantly from one evaluation to another, leading to uncertainty in the estimation of such values. Given the elitist nature of QD algorithms, they commonly end up with many degenerate solutions in such noisy settings. In this work, we introduce Archive Reproducibility Improvement Algorithm (ARIA); a plug-and-play approach that improves the reproducibility of the solutions present in an archive. We propose it as a separate optimization module, relying on natural evolution strategies, that can be executed on top of any QD algorithm. Our module mutates solutions to (1) optimize their probability of belonging to their niche, and (2) maximize their fitness. The performance of our method is evaluated on various tasks, including a classical optimization problem and two high-dimensional control tasks in simulated robotic environments. We show that our algorithm enhances the quality and descriptor space coverage of any given archive by at least 50%.