mlsanityData to 14 Oct 2026

Distributional AGI Safety

arXiv 2512.16856 ↗First posted 18 Dec 2025cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

AI safety and alignment research has predominantly been focused on methods for safeguarding individual AI systems, resting on the assumption of an eventual emergence of a monolithic Artificial General Intelligence (AGI). The alternative AGI emergence hypothesis, where general capability levels are first manifested through coordination in groups of sub-AGI individual agents with complementary skills and affordances, has received far less attention. Here we argue that this patchwork AGI hypothesis needs to be given serious consideration, and should inform the development of corresponding safeguards and mitigations. The rapid deployment of advanced AI agents with tool-use capabilities and the ability to communicate and coordinate makes this an urgent safety consideration. We therefore propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents. This framework centres on the design and implementation of virtual agentic sandbox economies (impermeable or semi-permeable), where agent-to-agent transactions are governed by robust market mechanisms, coupled with appropriate auditability, reputation management, and oversight to mitigate collective risks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 110 of its 199 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

110 of 199 references.

References · 110

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
38 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  3. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  4. 2022Toy Models of Superpositioncited by 971
  5. 2022In-context Learning and Induction Headscited by 964
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  8. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  9. 2024Self-Rewarding Language Modelscited by 681
  10. 2024A Survey on Large Language Models for Code Generationcited by 658
  11. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  13. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  14. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  15. 2020Artificial Intelligence, Values and Alignmentcited by 448
  16. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  17. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  18. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  19. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  20. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  21. 2023Tool Learning with Foundation Modelscited by 302
  22. 2023AI Alignment: A Comprehensive Surveycited by 263
  23. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  24. 2023PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimizationcited by 233
  25. 2023An Overview of Catastrophic AI Riskscited by 222
  26. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 213
  27. 2025Open Problems in Mechanistic Interpretabilitycited by 179
  28. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  29. 2020Open Problems in Cooperative AIcited by 168
  30. 2024The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Surveycited by 168
  31. 2025Multi-Agent Risks from Advanced AIcited by 163
  32. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  33. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  34. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  35. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 122
  36. 2017Emergent Complexity via Multi-Agent Competitioncited by 120
  37. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  38. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
  2. 2026Intelligent AI Delegationcited by 18
  3. 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defensescited by 7
  4. 2026Emergent Risks in Generative Multi-Agent Systemscited by 5
  5. 2026A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarmscited by 5
  6. 2026Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agentscited by 4
  7. 2026Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanismscited by 4
  8. 2026From AGI to ASIcited by 4
  9. 2026From Logic Monopoly to Social Contract: Separation of Power and the Institutional Foundations for Autonomous Agent Economiescited by 3
  10. 2026AI Cap-and-Trade: Efficiency Incentives for Accessibility and Sustainabilitycited by 0
  11. 2026The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systemscited by 0
  12. 2026Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Institutional AI: A Governance Framework for Distributional AGI Safety
  2. 2025Toward a Safe Internet of Agents
  3. 2025A Safety and Security Framework for Real-World Agentic Systems
  4. 2026Beyond single-channel agentic benchmarking
  5. 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agents
  6. 2025With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systems
  7. 2024From homeostasis to resource sharing: Biologically and economically aligned multi-objective multi-agent gridworld-based AI safety benchmarks
  8. 2025Alignment, Agency and Autonomy in Frontier AI: A Systems Engineering Perspective