# Selection-Expansion: A Unifying Framework for Motion-Planning and Diversity Search Algorithms

arXiv: https://arxiv.org/abs/2104.04768 · HTML: https://mlsanity.com/paper/2104.04768/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 10 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/1712.06567/) (2017)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors](https://mlsanity.com/paper/1905.11874/) (2019)
- [GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1802.05054/) (2018)
- [The problem with DDPG: understanding failures in deterministic environments with sparse rewards](https://mlsanity.com/paper/1911.11679/) (2019)
- [Safe Mutations for Deep and Recurrent Neural Networks through Output Gradients](https://mlsanity.com/paper/1712.06563/) (2017)
- [PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning](https://mlsanity.com/paper/2004.11667/) (2020)
- [Probabilistic completeness of RRT for geometric and kinodynamic planning with forward propagation](https://mlsanity.com/paper/1809.07051/) (2018)
- [Novelty Search makes Evolvability Inevitable](https://mlsanity.com/paper/2005.06224/) (2020)

## Built on by (most cited first, 1 of 1)

- [Density Descent for Diversity Optimization](https://mlsanity.com/paper/2312.11331/) (2023)

## Abstract (arXiv)

Reinforcement learning agents need a reward signal to learn successful policies. When this signal is sparse or the corresponding gradient is deceptive, such agents need a dedicated mechanism to efficiently explore their search space without relying on the reward. Looking for a large diversity of behaviors or using Motion Planning (MP) algorithms are two options in this context. In this paper, we build on the common roots between these two options to investigate the properties of two diversity search algorithms, the Novelty Search and the Goal Exploration Process algorithms. These algorithms look for diversity in an outcome space or behavioral space which is generally hand-designed to represent what matters for a given task. The relation to MP algorithms reveals that the smoothness, or lack of smoothness of the mapping between the policy parameter space and the outcome space plays a key role in the search efficiency. In particular, we show empirically that, if the mapping is smooth enough, i.e. if two close policies in the parameter space lead to similar outcomes, then diversity algorithms tend to inherit exploration properties of MP algorithms. By contrast, if it is not, diversity algorithms lose these properties and their performance strongly depends on specific heuristics, notably filtering mechanisms that discard some of the explored policies.