mlsanityData to 7 Oct 2026

Solipsistic Superintelligence is Unlikely to be Cooperative

arXiv 2606.03237 ↗First posted 2 Jun 2026cs.AI cs.CL cs.CYcited by 1

Not on the map. Why?

Abstract (arXiv)

AI's central challenge is shifting from capability to coexistence. The dominant paradigm in AI research focuses on developing powerful agents that treat the world as an exogenous and stationary source of feedback. We contend that superintelligence, an extremely capable task solver, born out of such a solipsistic approach to AI design, is unlikely to be cooperative. Deploying AI systems induces endogenous non-stationarity, resulting in a train-test-deploy gap where historical distributions diverge from the deployment context. We refer to this as the self-undermining property of unilateral optimization. Closing this gap requires AI that participates in cooperation: the equilibrium-selection process through which multiple actors navigate their interdependence. We call for a non-solipsistic research paradigm that treats this interdependence as a core design principle rather than approaching cooperation as a task to solve. This entails building dynamic evaluation testbeds involving adaptive counterparties, treating institutions as design primitives, and preserving human agency as a structural feature of the systems we build.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 53 of its 144 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

53 of 144 references.

References · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2020Performative Predictioncited by 318
  5. 2024Alignment faking in large language modelscited by 316
  6. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  7. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2023Playing repeated games with Large Language Modelscited by 213
  10. 2020Open Problems in Cooperative AIcited by 167
  11. 2021Alignment of Language Agentscited by 167
  12. 2023Model evaluation for extreme riskscited by 167
38 more
  1. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  2. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  3. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 134
  4. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  5. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  6. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  7. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 98
  8. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  9. 2017A Survey of Learning in Multiagent Environments: Dealing with Non-Stationaritycited by 93
  10. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  11. 2017A multi-agent reinforcement learning model of common-pool resource appropriationcited by 68
  12. 2021The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writerscited by 61
  13. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  14. 2024Algorithmic Collusion by Large Language Modelscited by 54
  15. 2019Auditing Radicalization Pathways on YouTubecited by 52
  16. 2025Gradual Disempowerment: Systemic Existential Risks from Incremental AI Developmentcited by 42
  17. 2023Position: AI Evaluation Should Learn from How We Test Humanscited by 36
  18. 2023Engagement, User Satisfaction, and the Amplification of Divisive Content on Social Mediacited by 30
  19. 2025Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Modelscited by 29
  20. 2024Empirical evidence of Large Language Model's influence on human spoken communicationcited by 27
  21. 2024Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitionscited by 25
  22. 2024A theory of appropriateness with applications to generative artificial intelligencecited by 25
  23. 2026Agents of Chaoscited by 24
  24. 2015Econometrics for Learning Agentscited by 23
  25. 2019The Role of Cooperation in Responsible AI Developmentcited by 21
  26. 2022Artificial Intelligence and Auction Designcited by 20
  27. 2024A social path to human-like artificial intelligencecited by 20
  28. 2026How LLMs Distort Our Written Languagecited by 20
  29. 2026OpenClaw Agents on Moltbook: Risky Instruction Sharing and Norm Enforcement in an Agent-Only Social Networkcited by 19
  30. 2021Adaptive Incentive Design with Multi-Agent Meta-Gradient Reinforcement Learningcited by 18
  31. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 18
  32. 2025Why Do Some Language Models Fake Alignment While Others Don't?cited by 16
  33. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
  34. 2025A Pragmatic View of AI Personhoodcited by 12
  35. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  36. 2025Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quiltcited by 9
  37. 2025Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Valuecited by 6
  38. 2026Agentic AI and the next intelligence explosioncited by 6

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.