Safety Must Precede the Deployment of Open-Ended AI
Not on the map. Why?
Abstract (arXiv)
AI advancements have been significantly driven by a combination of foundation models and curiosity-driven learning aimed at increasing capability and adaptability. Within this landscape, open-endedness, where AI agents autonomously and indefinitely generate novel behaviors, representations, or solutions, has gained increasing interest. This has become relevant in the context of self-evolving agents and long-horizon discovery. This position paper argues that the defining properties of open-ended AI systems introduce a distinct and underexplored class of safety challenges, including loss of predictability, emergent misalignment, and difficulties in maintaining effective control as systems evolve beyond their initial design assumptions, that must be addressed preemptively. These challenges differ qualitatively from those associated with task-bounded or static models and are unlikely to be addressed by existing safety frameworks alone, which is why these risks must be examined proactively, before large-scale deployment. The paper proposes a taxonomy for key challenges, discusses research opportunities, and calls for coordinated action to support the safe and responsible development of open-ended AI.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 68 of its 123 references that are in the map. First posted on arXiv 6 Feb 2025.
Builds on
68 of 123 references.
References · 68
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
38 more
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
- 2024Genie: Generative Interactive Environmentscited by 706
- 2018AI safety via debatecited by 384
- 2024Automated Design of Agentic Systemscited by 360
- 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
- 2018Reward learning from human preferences and demonstrations in Ataricited by 314
- 2021Unsolved Problems in ML Safetycited by 291
- 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
- 2023An Overview of Catastrophic AI Riskscited by 214
- 2024Evolutionary Optimization of Model Merging Recipescited by 204
- 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
- 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
- 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
- 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
- 2018Supervising strong learners by amplifying weak expertscited by 139
- 2022Evolution through Large Modelscited by 131
- 2022Language Models as Agent Modelscited by 131
- 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
- 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
- 2023Large Language Models as Evolutionary Optimizerscited by 122
- 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
- 2024Language Models Learn to Mislead Humans via RLHFcited by 118
- 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
- 2023Levels of AGI for Operationalizing Progress on the Path to AGIcited by 94
- 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 86
- 2019AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligencecited by 84
- 2024Robust agents learn causal world modelscited by 81
- 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
- 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72
- 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
- 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 60
- 2023Evaluating Superhuman Models with Consistency Checkscited by 47
- 2023Quality-Diversity through AI Feedbackcited by 40
- 2018Penalizing side effects using stepwise relative reachabilitycited by 38
- 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
- 2024Gradient based Feature Attribution in Explainable AI: A Technical Reviewcited by 34
- 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
- 2024Aligning Crowd Feedback via Distributional Preference Reward Modelingcited by 26
50 most cited of 68 All 68 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.