# Curiosity creates Diversity in Policy Search

arXiv: https://arxiv.org/abs/2212.03530 · HTML: https://mlsanity.com/paper/2212.03530/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 7 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents](https://mlsanity.com/paper/1712.06560/) (2017)
- [A survey on intrinsic motivation in reinforcement learning](https://mlsanity.com/paper/1908.06976/) (2019)
- [Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space](https://mlsanity.com/paper/1912.02400/) (2019)
- [Adaptive Stress Testing for Autonomous Vehicles](https://mlsanity.com/paper/1902.01909/) (2019)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors](https://mlsanity.com/paper/1905.11874/) (2019)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [EX2: Exploration with Exemplar Models for Deep Reinforcement Learning](https://mlsanity.com/paper/1703.01260/) (2017)
- [EvoJAX: Hardware-Accelerated Neuroevolution](https://mlsanity.com/paper/2202.05008/) (2022)
- [Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari](https://mlsanity.com/paper/1802.08842/) (2018)
- [Meta-learning curiosity algorithms](https://mlsanity.com/paper/2003.05325/) (2020)
- [Unsupervised Learning and Exploration of Reachable Outcome Space](https://mlsanity.com/paper/1909.05508/) (2019)
- [Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey](https://mlsanity.com/paper/2203.14009/) (2022)
- [Sparse Reward Exploration via Novelty Search and Emitters](https://mlsanity.com/paper/2102.03140/) (2021)
- [PNS: Population-Guided Novelty Search for Reinforcement Learning in Hard Exploration Environments](https://mlsanity.com/paper/1811.10264/) (2018)
- [ARCH-Elites: Quality-Diversity for Urban Design](https://mlsanity.com/paper/2104.08774/) (2021)

## Abstract (arXiv)

When searching for policies, reward-sparse environments often lack sufficient information about which behaviors to improve upon or avoid. In such environments, the policy search process is bound to blindly search for reward-yielding transitions and no early reward can bias this search in one direction or another. A way to overcome this is to use intrinsic motivation in order to explore new transitions until a reward is found. In this work, we use a recently proposed definition of intrinsic motivation, Curiosity, in an evolutionary policy search method. We propose Curiosity-ES, an evolutionary strategy adapted to use Curiosity as a fitness metric. We compare Curiosity with Novelty, a commonly used diversity metric, and find that Curiosity can generate higher diversity over full episodes without the need for an explicit diversity criterion and lead to multiple policies which find reward.