mlsanityData to 14 Oct 2026

Open-Endedness is Essential for Artificial Superhuman Intelligence

arXiv 2406.04268 ↗First posted 6 Jun 2024cs.LG cs.AIcited by 80

Not on the map. Why?

Abstract (arXiv)

In recent years there has been a tremendous surge in the general capabilities of AI systems, mainly fuelled by training foundation models on internetscale data. Nevertheless, the creation of openended, ever self-improving AI remains elusive. In this position paper, we argue that the ingredients are now in place to achieve openendedness in AI systems with respect to a human observer. Furthermore, we claim that such open-endedness is an essential property of any artificial superhuman intelligence (ASI). We begin by providing a concrete formal definition of open-endedness through the lens of novelty and learnability. We then illustrate a path towards ASI via open-ended systems built on top of foundation models, capable of making novel, humanrelevant discoveries. We conclude by examining the safety implications of generally-capable openended AI. We expect that open-ended foundation models will prove to be an increasingly fertile and safety-critical area of research in the near future.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 89 of its 147 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

89 of 147 references.

References · 89

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
38 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2018Exploration by Random Network Distillationcited by 993
  3. 2023Teaching Large Language Models to Self-Debugcited by 988
  4. 2023Large Language Models as Optimizerscited by 864
  5. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2024Self-Rewarding Language Modelscited by 681
  8. 2022Large Language Models Can Self-Improvecited by 664
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  12. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  13. 2023Learning Interactive Real-World Simulatorscited by 458
  14. 2015Illuminating search spaces by mapping elitescited by 425
  15. 2022Self-critiquing models for assisting human evaluatorscited by 380
  16. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 316
  17. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  18. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  19. 2016Cooperative Inverse Reinforcement Learningcited by 300
  20. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  21. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  22. 2023Language Modeling Is Compressioncited by 225
  23. 2020Prioritized Level Replaycited by 198
  24. 2023Large Language Models as General Pattern Machinescited by 197
  25. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 193
  26. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  27. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  28. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  29. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  30. 2023Vision-Language Models as Success Detectorscited by 142
  31. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  32. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  33. 2022Evolution through Large Modelscited by 134
  34. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  35. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  36. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  37. 2023Large Language Models as Evolutionary Optimizerscited by 124
  38. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 39; 2026: 32.

80 papers.

Sort:

By area:

  1. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 356
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  3. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  4. 2025Agentic Large Language Models, a surveycited by 108
  5. 2025A Self-Improving Coding Agentcited by 79
  6. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  7. 2026Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharingcited by 36
  8. 2024Towards evaluations-based safety cases for AI schemingcited by 33
  9. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  10. 2025Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesiscited by 22
  11. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  12. 2025HeuriGym: An Agentic Benchmark for LLM-Crafted Heuristics in Combinatorial Optimizationcited by 21
38 more
  1. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  2. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 19
  3. 2025Truly Self-Improving Agents Require Intrinsic Metacognitive Learningcited by 15
  4. 2026On Safety Risks in Experience-Driven Self-Evolving Agentscited by 14
  5. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 13
  6. 2024Boundless Socratic Learning with Language Gamescited by 12
  7. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  8. 2025Evolution and The Knightian Blindspot of Machine Learningcited by 11
  9. 2026Recursive self-improvement of AI research agentscited by 11
  10. 2025Automated Capability Discovery via Foundation Model Self-Explorationcited by 10
  11. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 9
  12. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
  13. 2025Intrinsically-Motivated Humans and Agents in Open-World Explorationcited by 8
  14. 2025OpenSIR: Open-Ended Self-Improving Reasonercited by 7
  15. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
  16. 2025ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamworkcited by 6
  17. 2026PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVRcited by 6
  18. 2026Stateless Yet Not Forgetful: Implicit Memory as a Hidden Channel in LLMscited by 6
  19. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  20. 2026OctoT2I: A Self-Evolving Agentic Text-to-Image Routercited by 5
  21. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  22. 2025Scalable Strategies for Continual Learning with Replaycited by 4
  23. 2025Curious Causality-Seeking Agents Learn Meta Causal Worldcited by 4
  24. 2025Safety Must Precede the Deployment of Open-Ended AIcited by 3
  25. 2025Measuring In-Context Computation Complexity via Hidden State Predictioncited by 3
  26. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  27. 2025Policy Compatible Skill Incremental Learning via Lazy Learning Interfacecited by 3
  28. 2026KellyBench: A Benchmark for Long-Horizon Sequential Decision Makingcited by 3
  29. 2026Interestingness as an Inductive Heuristic for Future Compression Progresscited by 3
  30. 2026What Do Evolutionary Coding Agents Evolve?cited by 3
  31. 2026Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontiercited by 3
  32. 2024Incentives to Build Houses, Trade Houses, or Trade House Building Skills in Simulated Worlds under Various Governing Systems or Institutions: Comparing Multi-agent Reinforcement Learning to Generative Agent-based Modelcited by 2
  33. 2025Humans Coexist, So Must Embodied Artificial Agentscited by 2
  34. 2025Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measurescited by 2
  35. 2025To Trade or Not to Trade: An Agentic Approach to Estimating Market Risk Improves Trading Decisionscited by 2
  36. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 2
  37. 2025On Improvisation and Open-Endedness: Insights for Experiential AIcited by 2
  38. 2026The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safetycited by 2

50 most cited of 80 All 80 (JSON)