# Unsupervised Learning of Goal Spaces for Intrinsically Motivated Goal Exploration

arXiv: https://arxiv.org/abs/1803.00781 · HTML: https://mlsanity.com/paper/1803.00781/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2018.

## Built on by (most cited first, 23 of 23)

- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Offline Meta-Reinforcement Learning with Online Self-Supervision](https://mlsanity.com/paper/2107.03974/) (2021)
- [Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2110.13625/) (2021)
- [Unsupervised Behaviour Discovery with Quality-Diversity Optimisation](https://mlsanity.com/paper/2106.05648/) (2021)
- [What Can I Do Here? Learning New Skills by Imagining Visual Affordances](https://mlsanity.com/paper/2106.00671/) (2021)
- [Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution](https://mlsanity.com/paper/2104.13424/) (2021)
- [Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires](https://mlsanity.com/paper/2109.08522/) (2021)
- [Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/2104.05043/) (2021)
- [Active Hierarchical Exploration with Stable Subgoal Representation Learning](https://mlsanity.com/paper/2105.14750/) (2021)
- [Self-supervised Reinforcement Learning with Independently Controllable Subgoals](https://mlsanity.com/paper/2109.04150/) (2021)
- [Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Vision](https://mlsanity.com/paper/2210.15206/) (2022)
- [Learning Domain Invariant Representations in Goal-conditioned Block MDPs](https://mlsanity.com/paper/2110.14248/) (2021)
- [Relevance-guided Unsupervised Discovery of Abilities with Quality-Diversity Algorithms](https://mlsanity.com/paper/2204.09828/) (2022)
- [Probabilistic Subgoal Representations for Hierarchical Reinforcement learning](https://mlsanity.com/paper/2406.16707/) (2024)
- [From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models](https://mlsanity.com/paper/2407.09502/) (2024)
- [Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learning](https://mlsanity.com/paper/2203.00874/) (2022)
- [Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs](https://mlsanity.com/paper/2307.12063/) (2023)
- [Life-inspired Interoceptive Artificial Intelligence for Autonomous and Adaptive Agents](https://mlsanity.com/paper/2309.05999/) (2023)
- [Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agents](https://mlsanity.com/paper/2406.12359/) (2024)
- [Sample Efficient Robot Learning in Supervised Effect Prediction Tasks](https://mlsanity.com/paper/2412.02331/) (2024)
- [Expedition & Expansion: Leveraging Semantic Representations for Goal-Directed Exploration in Continuous Cellular Automata](https://mlsanity.com/paper/2509.03863/) (2025)
- [A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorld](https://mlsanity.com/paper/2302.05244/) (2023)
- [Landmark Guided Active Exploration with State-specific Balance Coefficient](https://mlsanity.com/paper/2306.17484/) (2023)

## Abstract (arXiv)

Intrinsically motivated goal exploration algorithms enable machines to discover repertoires of policies that produce a diversity of effects in complex environments. These exploration algorithms have been shown to allow real world robots to acquire skills such as tool use in high-dimensional continuous state and action spaces. However, they have so far assumed that self-generated goals are sampled in a specifically engineered feature space, limiting their autonomy. In this work, we propose to use deep representation learning algorithms to learn an adequate goal space. This is a developmental 2-stage approach: first, in a perceptual learning stage, deep learning algorithms use passive raw sensor observations of world changes to learn a corresponding latent space; then goal exploration happens in a second stage by sampling goals in this latent space. We present experiments where a simulated robot arm interacts with an object, and we show that exploration algorithms using such learned representations can match the performance obtained using engineered representations.