mlsanityData to 7 Oct 2026

Open-Ended Learning Leads to Generally Capable Agents

arXiv 2107.12808 ↗First posted 27 Jul 2021cs.LG cs.AI cs.MAcited by 177

Not on the map. Why?

Cited by 28 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Builds on 56 of its 109 references that are in the map. First posted on arXiv 27 Jul 2021.

Builds on

56 of 109 references.

References · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2019Solving Rubik's Cube with a Robot Handcited by 699
  11. 2017Deep Mutual Learningcited by 621
  12. 2018Born Again Neural Networkscited by 539
38 more
  1. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  2. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  3. 2017Population Based Training of Neural Networkscited by 351
  4. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  5. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  6. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  7. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  8. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  9. 2017Automated Curriculum Learning for Neural Networkscited by 251
  10. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  11. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  12. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  13. 2017Teacher-Student Curriculum Learningcited by 197
  14. 2020Prioritized Level Replaycited by 190
  15. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  16. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  17. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  18. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  19. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  20. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  21. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  22. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  23. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  24. 2020PCGRL: Procedural Content Generation via Reinforcement Learningcited by 90
  25. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  26. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  27. 2019AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligencecited by 84
  28. 2019Distilling Policy Distillationcited by 81
  29. 2016Learning values across many orders of magnitudecited by 80
  30. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
  31. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  32. 2018Kickstarting Deep Reinforcement Learningcited by 75
  33. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  34. 2020Real World Games Look Like Spinning Topscited by 66
  35. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  36. 2018Re-evaluating Evaluationcited by 47
  37. 2020A Boolean Task Algebra for Reinforcement Learningcited by 46
  38. 2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 34

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 43; 2023: 46; 2024: 29; 2025: 32; 2026: 17.

177 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  5. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  6. 2023Augmented Language Models: a Surveycited by 398
  7. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  8. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  9. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  10. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  11. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  12. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
38 more
  1. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  2. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  3. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  4. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  5. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  6. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
  7. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  8. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  9. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  10. 2022Retrospectives on the Embodied AI Workshopcited by 46
  11. 2022A Survey of Ad Hoc Teamwork Researchcited by 41
  12. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  13. 2021Illuminating Diverse Neural Cellular Automata for Level Generationcited by 35
  14. 2023Skill Reinforcement Learning and Planning for Open-World Long-Horizon Taskscited by 35
  15. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
  16. 2025How do language models learn facts? Dynamics, curricula and hallucinationscited by 33
  17. 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 32
  18. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 31
  19. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  20. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  21. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  22. 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
  23. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  24. 2022Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learningcited by 27
  25. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  26. 2025Large Language Models Miss the Multi-Agent Markcited by 25
  27. 2023Creative Agents: Empowering Agents with Imagination for Creative Taskscited by 24
  28. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 23
  29. 2022Mind the gap: Challenges of deep learning approaches to Theory of Mindcited by 21
  30. 2022Bayesian Generational Population-Based Trainingcited by 20
  31. 2022General Intelligence Requires Rethinking Explorationcited by 20
  32. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  33. 2024A social path to human-like artificial intelligencecited by 20
  34. 2022Improving Policy Optimization with Generalist-Specialist Learningcited by 19
  35. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  36. 2024AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Makingcited by 19
  37. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 19
  38. 2021TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrationscited by 18

50 most cited of 177 All 177 (JSON)