# Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

arXiv: https://arxiv.org/abs/1708.02190 · HTML: https://mlsanity.com/paper/1708.02190/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Aug 2017.

## Built on by (most cited first, 30 of 83)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [A Survey of Exploration Methods in Reinforcement Learning](https://mlsanity.com/paper/2109.00157/) (2021)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [Adversarial Intrinsic Motivation for Reinforcement Learning](https://mlsanity.com/paper/2105.13345/) (2021)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation](https://mlsanity.com/paper/2210.12631/) (2022)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Causality-driven Hierarchical Structure Discovery for Reinforcement Learning](https://mlsanity.com/paper/2210.06964/) (2022)
- [CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models](https://mlsanity.com/paper/2409.18382/) (2024)
- [Mind the gap: Challenges of deep learning approaches to Theory of Mind](https://mlsanity.com/paper/2203.16540/) (2022)
- [Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoires](https://mlsanity.com/paper/2109.08522/) (2021)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [Self-supervised Reinforcement Learning with Independently Controllable Subgoals](https://mlsanity.com/paper/2109.04150/) (2021)
- [Eurekaverse: Environment Curriculum Generation via Large Language Models](https://mlsanity.com/paper/2411.01775/) (2024)
- [Sparse Reward Exploration via Novelty Search and Emitters](https://mlsanity.com/paper/2102.03140/) (2021)
- [Computational Theories of Curiosity-Driven Learning](https://mlsanity.com/paper/1802.10546/) (2018)
- [Imitation Learning from Observation with Automatic Discount Scheduling](https://mlsanity.com/paper/2310.07433/) (2023)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [minimax: Efficient Baselines for Autocurricula in JAX](https://mlsanity.com/paper/2311.12716/) (2023)
- [Goal-Guided Neural Cellular Automata: Learning to Control Self-Organising Systems](https://mlsanity.com/paper/2205.06806/) (2022)
- [Automatic Acquisition of a Repertoire of Diverse Grasping Trajectories through Behavior Shaping and Novelty Search](https://mlsanity.com/paper/2205.08189/) (2022)
- [Biomaker CA: a Biome Maker project using Cellular Automata](https://mlsanity.com/paper/2307.09320/) (2023)
- [Discovering Sensorimotor Agency in Cellular Automata using Diversity Search](https://mlsanity.com/paper/2402.10236/) (2024)
- [Position: Foundation Agents as the Paradigm Shift for Decision Making](https://mlsanity.com/paper/2405.17009/) (2024)
- [Training a First-Order Theorem Prover from Synthetic Data](https://mlsanity.com/paper/2103.03798/) (2021)
- [Efficient Self-Supervised Data Collection for Offline Robot Learning](https://mlsanity.com/paper/2105.04607/) (2021)
- [Few-shot Quality-Diversity Optimization](https://mlsanity.com/paper/2109.06826/) (2021)
- [From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models](https://mlsanity.com/paper/2407.09502/) (2024)
- [Don't Do What Doesn't Matter: Intrinsic Motivation with Action Usefulness](https://mlsanity.com/paper/2105.09992/) (2021)

## Abstract (arXiv)

Intrinsically motivated spontaneous exploration is a key enabler of autonomous developmental learning in human children. It enables the discovery of skill repertoires through autotelic learning, i.e. the self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present an algorithmic approach called Intrinsically Motivated Goal Exploration Processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP architecture relies on several principles: 1) self-generation of goals, generalized as parameterized fitness functions; 2) selection of goals based on intrinsic rewards; 3) exploration with incremental goal-parameterized policy search and exploitation with a batch learning algorithm; 4) systematic reuse of information acquired when targeting a goal for improving towards other goals. We present a particularly efficient form of IMGEP, called AMB, that uses a population-based policy and an object-centered spatio-temporal modularity. We provide several implementations of this architecture and demonstrate their ability to automatically generate a learning curriculum within several experimental setups. One of these experiments includes a real humanoid robot exploring multiple spaces of goals with several hundred continuous dimensions and with distractors. While no particular target goal is provided to these autotelic agents, this curriculum allows the discovery of diverse skills that act as stepping stones for learning more complex skills, e.g. nested tool use.