# Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution

arXiv: https://arxiv.org/abs/2104.13424 · HTML: https://mlsanity.com/paper/2104.13424/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 27 Apr 2021.

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Predicting Parameters in Deep Learning](https://mlsanity.com/paper/1306.0543/) (2013)
- [Convergent Learning: Do different neural networks learn the same representations?](https://mlsanity.com/paper/1511.07543/) (2015)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Using Fast Weights to Attend to the Recent Past](https://mlsanity.com/paper/1610.06258/) (2016)
- [Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning](https://mlsanity.com/paper/1712.06567/) (2017)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016)
- [One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers](https://mlsanity.com/paper/1906.02773/) (2019)
- [Weight Agnostic Neural Networks](https://mlsanity.com/paper/1906.04358/) (2019)
- [Procedural Content Generation through Quality Diversity](https://mlsanity.com/paper/1907.04053/) (2019)
- [Discovering the Elite Hypervolume by Leveraging Interspecies Correlation](https://mlsanity.com/paper/1804.03906/) (2018)
- [Scaling MAP-Elites to Deep Neuroevolution](https://mlsanity.com/paper/2003.01825/) (2020)
- [Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration](https://mlsanity.com/paper/1803.00781/) (2018)
- [Deep Fried Convnets](https://mlsanity.com/paper/1412.7149/) (2014)
- [Active Learning based on Data Uncertainty and Model Sensitivity](https://mlsanity.com/paper/1808.02026/) (2018)
- [Agent Embeddings: A Latent Representation for Pole-Balancing Networks](https://mlsanity.com/paper/1811.04516/) (2018)
- [Biologically inspired architectures for sample-efficient deep reinforcement learning](https://mlsanity.com/paper/1911.11285/) (2019)

## Built on by (most cited first, 20 of 20)

- [Differentiable Quality Diversity](https://mlsanity.com/paper/2106.03894/) (2021)
- [Deep Surrogate Assisted Generation of Environments](https://mlsanity.com/paper/2206.04199/) (2022)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2205.10752/) (2022)
- [On the Versatile Uses of Partial Distance Correlation in Deep Learning](https://mlsanity.com/paper/2207.09684/) (2022)
- [Generating Behaviorally Diverse Policies with Latent Diffusion Models](https://mlsanity.com/paper/2305.18738/) (2023)
- [COIL: Constrained Optimization in Learned Latent Space: Learning Representations for Valid Solutions](https://mlsanity.com/paper/2202.02163/) (2022)
- [The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformers](https://mlsanity.com/paper/2303.16207/) (2023)
- [Learning Robust Real-Time Cultural Transmission without Human Data](https://mlsanity.com/paper/2203.00715/) (2022)
- [Evolving Populations of Diverse RL Agents with MAP-Elites](https://mlsanity.com/paper/2303.12803/) (2023)
- [From Parameters to Behaviors: Unsupervised Compression of the Policy Space](https://mlsanity.com/paper/2509.22566/) (2025)
- [Generative Design through Quality-Diversity Data Synthesis and Language Models](https://mlsanity.com/paper/2405.09997/) (2024)
- [Rigorous Runtime Analysis of Diversity Optimization with GSEMO on OneMinMax](https://mlsanity.com/paper/2307.07248/) (2023)
- [Density Descent for Diversity Optimization](https://mlsanity.com/paper/2312.11331/) (2023)
- [Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spaces](https://mlsanity.com/paper/2601.01082/) (2026)
- [Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching](https://mlsanity.com/paper/2603.27044/) (2026)
- [Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning](https://mlsanity.com/paper/2604.15414/) (2026)
- [Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning](https://mlsanity.com/paper/2605.26012/) (2026)
- [Behavioral Foundation Models for Quality Diversity](https://mlsanity.com/paper/2609.35615/) (2026)

## Abstract (arXiv)

Neuroevolution is an alternative to gradient-based optimisation that has the potential to avoid local minima and allows parallelisation. The main limiting factor is that usually it does not scale well with parameter space dimensionality. Inspired by recent work examining neural network intrinsic dimension and loss landscapes, we hypothesise that there exists a low-dimensional manifold, embedded in the policy network parameter space, around which a high-density of diverse and useful policies are located. This paper proposes a novel method for diversity-based policy search via Neuroevolution, that leverages learned representations of the policy network parameters, by performing policy search in this learned representation space. Our method relies on the Quality-Diversity (QD) framework which provides a principled approach to policy search, and maintains a collection of diverse policies, used as a dataset for learning policy representations. Further, we use the Jacobian of the inverse-mapping function to guide the search in the representation space. This ensures that the generated samples remain in the high-density regions, after mapping back to the original space. Finally, we evaluate our contributions on four continuous-control tasks in simulated environments, and compare to diversity-based baselines.