# Unsupervised Learning and Exploration of Reachable Outcome Space

arXiv: https://arxiv.org/abs/1909.05508 · HTML: https://mlsanity.com/paper/1909.05508/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2019.

## Built on by (most cited first, 23 of 23)

- [Unsupervised Behaviour Discovery with Quality-Diversity Optimisation](https://mlsanity.com/paper/2106.05648/) (2021)
- [Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires](https://mlsanity.com/paper/2109.08522/) (2021)
- [Accelerated Quality-Diversity through Massive Parallelism](https://mlsanity.com/paper/2202.01258/) (2022)
- [Sparse Reward Exploration via Novelty Search and Emitters](https://mlsanity.com/paper/2102.03140/) (2021)
- [Automatic Acquisition of a Repertoire of Diverse Grasping Trajectories through Behavior Shaping and Novelty Search](https://mlsanity.com/paper/2205.08189/) (2022)
- [Relevance-guided Unsupervised Discovery of Abilities with Quality-Diversity Algorithms](https://mlsanity.com/paper/2204.09828/) (2022)
- [BR-NS: an Archive-less Approach to Novelty Search](https://mlsanity.com/paper/2104.03936/) (2021)
- [Learning to Walk Autonomously via Reset-Free Quality-Diversity](https://mlsanity.com/paper/2204.03655/) (2022)
- [Online Damage Recovery for Physical Robots with Hierarchical Quality-Diversity](https://mlsanity.com/paper/2210.09918/) (2022)
- [Quality Diversity under Sparse Reward and Sparse Interaction: Application to Grasping in Robotics](https://mlsanity.com/paper/2308.05483/) (2023)
- [Discovering and Exploiting Sparse Rewards in a Learned Behavior Space](https://mlsanity.com/paper/2111.01919/) (2021)
- [Hierarchical Quality-Diversity for Online Damage Recovery](https://mlsanity.com/paper/2204.05726/) (2022)
- [Generative Design through Quality-Diversity Data Synthesis and Language Models](https://mlsanity.com/paper/2405.09997/) (2024)
- [AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization](https://mlsanity.com/paper/2506.05634/) (2025)
- [Quality-Diversity Meta-Evolution: customising behaviour spaces to a meta-objective](https://mlsanity.com/paper/2109.03918/) (2021)
- [Discovering Unsupervised Behaviours from Full-State Trajectories](https://mlsanity.com/paper/2211.15451/) (2022)
- [Density Descent for Diversity Optimization](https://mlsanity.com/paper/2312.11331/) (2023)
- [Guided Safe Shooting: model based reinforcement learning with safety constraints](https://mlsanity.com/paper/2206.09743/) (2022)
- [Vector Quantized-Elites: Unsupervised and Problem-Agnostic Quality-Diversity Optimization](https://mlsanity.com/paper/2504.08057/) (2025)
- [Soft Quality-Diversity Optimization](https://mlsanity.com/paper/2512.00810/) (2025)
- [Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spaces](https://mlsanity.com/paper/2601.01082/) (2026)
- [Curiosity creates Diversity in Policy Search](https://mlsanity.com/paper/2212.03530/) (2022)
- [From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversity](https://mlsanity.com/paper/2508.19172/) (2025)

## Abstract (arXiv)

Performing Reinforcement Learning in sparse rewards settings, with very little prior knowledge, is a challenging problem since there is no signal to properly guide the learning process. In such situations, a good search strategy is fundamental. At the same time, not having to adapt the algorithm to every single problem is very desirable. Here we introduce TAXONS, a Task Agnostic eXploration of Outcome spaces through Novelty and Surprise algorithm. Based on a population-based divergent-search approach, it learns a set of diverse policies directly from high-dimensional observations, without any task-specific information. TAXONS builds a repertoire of policies while training an autoencoder on the high-dimensional observation of the final state of the system to build a low-dimensional outcome space. The learned outcome space, combined with the reconstruction error, is used to drive the search for new policies. Results show that TAXONS can find a diverse set of controllers, covering a good part of the ground-truth outcome space, while having no information about such space.