# A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents

arXiv: https://arxiv.org/abs/2311.00344 · HTML: https://mlsanity.com/paper/2311.00344/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 64 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions](https://mlsanity.com/paper/2003.08536/) (2020)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [A Definition of Continual Reinforcement Learning](https://mlsanity.com/paper/2307.11046/) (2023)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Abstraction for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.05839/) (2022)
- [Learning to Generate Levels From Nothing](https://mlsanity.com/paper/2002.05259/) (2020)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)
- [Co-generation of game levels and game-playing agents](https://mlsanity.com/paper/2007.08497/) (2020)
- [An Overview of Open-Ended Evolution: Editorial Introduction to the Open-Ended Evolution II Special Issue](https://mlsanity.com/paper/1909.04430/) (2019)
- [Watts: Infrastructure for Open-Ended Learning](https://mlsanity.com/paper/2204.13250/) (2022)
- [Autonomous discovery of the goal space to learn a parameterized skill](https://mlsanity.com/paper/1805.07547/) (2018)

## Built on by (most cited first, 18 of 18)

- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [BuilderBench: The Building Blocks of Intelligent Agents](https://mlsanity.com/paper/2510.06288/) (2025)
- [Safety Must Precede the Deployment of Open-Ended AI](https://mlsanity.com/paper/2502.04512/) (2025)
- [Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures](https://mlsanity.com/paper/2506.06832/) (2025)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Polaris: A G"odel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair](https://mlsanity.com/paper/2603.23129/) (2026)
- [The Autonomy-Alignment Problem in Open-Ended Learning Robots: Formalising the Purpose Framework](https://mlsanity.com/paper/2403.02514/) (2024)
- [A Motivational Architecture for Open-Ended Learning Challenges in Robots](https://mlsanity.com/paper/2506.18454/) (2025)
- [Improving Zero-Shot Offline RL via Behavioral Task Sampling](https://mlsanity.com/paper/2604.25496/) (2026)
- [Synthesizing Evolving Symbolic Representations for Autonomous Systems](https://mlsanity.com/paper/2409.11756/) (2024)
- [Focusing Robot Open-Ended Reinforcement Learning Through Users' Purposes](https://mlsanity.com/paper/2503.12579/) (2025)
- [Autonomous state-space segmentation for Deep-RL sparse reward scenarios](https://mlsanity.com/paper/2504.03420/) (2025)
- [HERAKLES: Hierarchical Skill Compilation for Open-ended LLM Agents](https://mlsanity.com/paper/2508.14751/) (2025)
- [Discovering Novel LLM Experts via Task-Capability Coevolution](https://mlsanity.com/paper/2604.14969/) (2026)
- [An Information-Theoretic Definition for Open-Ended Learning](https://mlsanity.com/paper/2606.08369/) (2026)
- [Acceptable Misalignment](https://mlsanity.com/paper/2610.16881/) (2026)

## Abstract (arXiv)

A lot of recent machine learning research papers have ``open-ended learning'' in their title. But very few of them attempt to define what they mean when using the term. Even worse, when looking more closely there seems to be no consensus on what distinguishes open-ended learning from related concepts such as continual learning, lifelong learning or autotelic learning. In this paper, we contribute to fixing this situation. After illustrating the genealogy of the concept and more recent perspectives about what it truly means, we outline that open-ended learning is generally conceived as a composite notion encompassing a set of diverse properties. In contrast with previous approaches, we propose to isolate a key elementary property of open-ended processes, which is to produce elements from time to time (e.g., observations, options, reward functions, and goals), over an infinite horizon, that are considered novel from an observer's perspective. From there, we build the notion of open-ended learning problems and focus in particular on the subset of open-ended goal-conditioned reinforcement learning problems in which agents can learn a growing repertoire of goal-driven skills. Finally, we highlight the work that remains to be performed to fill the gap between our elementary definition and the more involved notions of open-ended learning that developmental AI researchers may have in mind.