# Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

arXiv: https://arxiv.org/abs/2003.08536 · HTML: https://mlsanity.com/paper/2003.08536/

## Key facts

- Cited by 106 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 30 of 106)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Automated Design of Agentic Systems](https://mlsanity.com/paper/2408.08435/) (2024)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [CAT: Closed-loop Adversarial Training for Safe End-to-End Driving](https://mlsanity.com/paper/2310.12432/) (2023)
- [Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Tasks](https://mlsanity.com/paper/2410.23208/) (2024)
- [Deep Surrogate Assisted Generation of Environments](https://mlsanity.com/paper/2206.04199/) (2022)
- [Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft](https://mlsanity.com/paper/2106.14876/) (2021)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [Augmenting Autotelic Agents with Large Language Models](https://mlsanity.com/paper/2305.12487/) (2023)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems](https://mlsanity.com/paper/2111.04613/) (2021)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis](https://mlsanity.com/paper/2505.11581/) (2025)
- [Mind the gap: Challenges of deep learning approaches to Theory of Mind](https://mlsanity.com/paper/2203.16540/) (2022)
- [Bayesian Generational Population-Based Training](https://mlsanity.com/paper/2207.09405/) (2022)
- [CLUTR: Curriculum Learning via Unsupervised Task Representation Learning](https://mlsanity.com/paper/2210.10243/) (2022)
- [General Intelligence Requires Rethinking Exploration](https://mlsanity.com/paper/2211.07819/) (2022)
- [Automating the Search for Artificial Life with Foundation Models](https://mlsanity.com/paper/2412.17799/) (2024)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents](https://mlsanity.com/paper/2311.00344/) (2023)

## Abstract (arXiv)

Creating open-ended algorithms, which generate their own never-ending stream of novel and appropriately challenging learning opportunities, could help to automate and accelerate progress in machine learning. A recent step in this direction is the Paired Open-Ended Trailblazer (POET), an algorithm that generates and solves its own challenges, and allows solutions to goal-switch between challenges to avoid local optima. However, the original POET was unable to demonstrate its full creative potential because of limitations of the algorithm itself and because of external issues including a limited problem space and lack of a universal progress measure. Importantly, both limitations pose impediments not only for POET, but for the pursuit of open-endedness in general. Here we introduce and empirically validate two new innovations to the original algorithm, as well as two external innovations designed to help elucidate its full potential. Together, these four advances enable the most open-ended algorithmic demonstration to date. The algorithmic innovations are (1) a domain-general measure of how meaningfully novel new challenges are, enabling the system to potentially create and solve interesting challenges endlessly, and (2) an efficient heuristic for determining when agents should goal-switch from one problem to another (helping open-ended search better scale). Outside the algorithm itself, to enable a more definitive demonstration of open-endedness, we introduce (3) a novel, more flexible way to encode environmental challenges, and (4) a generic measure of the extent to which a system continues to exhibit open-ended innovation. Enhanced POET produces a diverse range of sophisticated behaviors that solve a wide range of environmental challenges, many of which cannot be solved through other means.