Solipsistic Superintelligence is Unlikely to be Cooperative
Not on the map. Why?
Abstract (arXiv)
AI's central challenge is shifting from capability to coexistence. The dominant paradigm in AI research focuses on developing powerful agents that treat the world as an exogenous and stationary source of feedback. We contend that superintelligence, an extremely capable task solver, born out of such a solipsistic approach to AI design, is unlikely to be cooperative. Deploying AI systems induces endogenous non-stationarity, resulting in a train-test-deploy gap where historical distributions diverge from the deployment context. We refer to this as the self-undermining property of unilateral optimization. Closing this gap requires AI that participates in cooperation: the equilibrium-selection process through which multiple actors navigate their interdependence. We call for a non-solipsistic research paradigm that treats this interdependence as a core design principle rather than approaching cooperation as a task to solve. This entails building dynamic evaluation testbeds involving adaptive counterparties, treating institutions as design primitives, and preserving human agency as a structural feature of the systems we build.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 53 of its 144 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
53 of 144 references.
References · 53
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2020Performative Predictioncited by 318
- 2024Alignment faking in large language modelscited by 316
- 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
- 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
- 2023Playing repeated games with Large Language Modelscited by 213
- 2020Open Problems in Cooperative AIcited by 167
- 2021Alignment of Language Agentscited by 167
- 2023Model evaluation for extreme riskscited by 167
38 more
- 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
- 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
- 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
- 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
- 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
- 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
- 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
- 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
- 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
- 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
- 2017A multi-agent reinforcement learning model of common-pool resource appropriationcited by 68
- 2021The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writerscited by 61
- 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
- 2024Algorithmic Collusion by Large Language Modelscited by 54
- 2019Auditing Radicalization Pathways on YouTubecited by 52
- 2025Gradual Disempowerment: Systemic Existential Risks from Incremental AI Developmentcited by 42
- 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
- 2023Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Mediacited by 30
- 2025Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Modelscited by 29
- 2024Empirical evidence of Large Language Model's influence on human spoken communicationcited by 27
- 2024Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitionscited by 25
- 2024A theory of appropriateness with applications to generative artificial intelligencecited by 25
- 2026Agents of Chaoscited by 24
- 2015Econometrics for Learning Agentscited by 23
- 2019The Role of Cooperation in Responsible AI Developmentcited by 21
- 2022Artificial Intelligence and Auction Designcited by 20
- 2024A social path to human-like artificial intelligencecited by 20
- 2026How LLMs Distort Our Written Languagecited by 20
- 2026OpenClaw Agents on Moltbook: Risky Instruction Sharing and Norm Enforcement in an Agent-Only Social Networkcited by 19
- 2021Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learningcited by 18
- 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
- 2025Why Do Some Language Models Fake Alignment While Others Don't?cited by 16
- 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
- 2025A Pragmatic View of AI Personhoodcited by 12
- 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
- 2025Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quiltcited by 9
- 2025Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Valuecited by 6
- 2026Agentic AI and the next intelligence explosioncited by 6
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.