mlsanityData to 7 Oct 2026

Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs

arXiv 2601.11369 ↗First posted 16 Jan 2026cs.GT cs.AIcited by 17

On the map: LLM agents and tool use → Financial trading and forecasting agents

Abstract (arXiv)

Multi-agent LLM ensembles can converge on coordinated, socially harmful equilibria. This paper advances an experimental framework for evaluating Institutional AI, our system-level approach to AI alignment that reframes alignment from preference engineering in agent-space to mechanism design in institution-space. Central to this approach is the governance graph, a public, immutable manifest that declares legal states, transitions, sanctions, and restorative paths; an Oracle/Controller runtime interprets this manifest, attaching enforceable consequences to evidence of coordination while recording a cryptographically keyed, append-only governance log for audit and provenance. We apply the Institutional AI framework to govern the Cournot collusion case documented by prior work and compare three regimes: Ungoverned (baseline incentives from the structure of the Cournot market), Constitutional (a prompt-only policy-as-prompt prohibition implemented as a fixed written anti-collusion constitution, and Institutional (governance-graph-based). Across six model configurations including cross-provider pairs (N=90 runs/condition), the Institutional regime produces large reductions in collusion: mean tier falls from 3.1 to 1.8 (Cohen's d=1.28), and severe-collusion incidence drops from 50% to 5.6%. The prompt-only Constitutional baseline yields no reliable improvement, illustrating that declarative prohibitions do not bind under optimisation pressure. These results suggest that multi-agent alignment may benefit from being framed as an institutional design problem, where governance graphs can provide a tractable abstraction for alignment-relevant collective behavior.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Financial trading and forecasting agents, one of 16 directions in LLM agents and tool use. Builds on 41 of its 85 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

41 of 85 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Evaluating LLM Agent Collusion in Double Auctions2 Jul
  2. 2024Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitions19 Sep

Other references · 39

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  4. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  5. 2024Alignment faking in large language modelscited by 316
  6. 2024Frontier Models are Capable of In-context Schemingcited by 252
  7. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  8. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  9. 2025Multi-Agent Risks from Advanced AIcited by 150
  10. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 120
  11. 2024LLM Multi-Agent Systems: Challenges and Open Problemscited by 119
  12. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
27 more
  1. 2019Optimal Policies Tend to Seek Powercited by 115
  2. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  3. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 78
  4. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  5. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agentscited by 62
  6. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  7. 2025Characterizing AI Agents for Alignment and Governancecited by 30
  8. 2025Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Modelscited by 29
  9. 2024Network Formation and Dynamics Among Multi-LLMscited by 28
  10. 2025Emergent Coordination in Multi-Agent Language Modelscited by 22
  11. 2025Virtual Agent Economiescited by 17
  12. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 16
  13. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  14. 2025Beyond Single-Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactionscited by 10
  15. 2025Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMscited by 8
  16. 2025Distributional AGI Safetycited by 8
  17. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 7
  18. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  19. 2024Roadmap on Incentive Compatibility for AI Alignment and Governance in Sociotechnical Systemscited by 6
  20. 2026Institutional AI: A Governance Framework for Distributional AGI Safetycited by 6
  21. 2025AI Governance through Marketscited by 5
  22. 2019No-regret Learning in Cournot Gamescited by 4
  23. 2020Multi-Agent Reinforcement Learning in Cournot Gamescited by 4
  24. 2026Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systemscited by 4
  25. 2024Emergent Cooperation under Uncertain Incentive Alignmentcited by 3
  26. 2026When Numbers Start Talking: Implicit Numerical Coordination Among LLM-Based Agentscited by 2
  27. 2026Vertical tacit collusion in AI-mediated marketscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Detecting Multi-Agent Collusion Through Multi-Agent Interpretabilitycited by 6
  2. 2026Organizational Control Layer: Governance Infrastructure at the Execution Boundary of LLM Agent Systemscited by 3
  3. 2026Agentic Microphysics: A Manifesto for Generative AI Safetycited by 2
  4. 2026Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safetycited by 1
  5. 2026Draining the Energy Commons: Self-Defeating Over-Appropriation as a Coordination Failure in Agentic LLM Collectivescited by 1
  6. 2026Multi-Agent AI Safety as an Institutional Design Problemcited by 1
  7. 2026Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisionscited by 1
  8. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1
  9. 2026Structural Rigidity and the 57-Token Predictive Window: A Physical Framework for Inference-Layer Governability in Large Language Modelscited by 0
  10. 2026The Cognitive Penalty: Ablating System 1 and System 2 Reasoning in Edge-Native SLMs for Decentralized Consensuscited by 0
  11. 2026Mechanical Enforcement for LLM Governance:Evidence of Governance-Task Decoupling in Financial Decision Systemscited by 0
  12. 2026Formal Mechanisms for Market Stability in Self-Interested Agent Societies: A Marketplace Simulation Studycited by 0
5 more
  1. 2026Where Reliability Lives: Experimental Localisation of Behavioural Properties in an Agent Systemcited by 0
  2. 2026WolfSociety: Understanding Collective Risk from Harmful-Agent Scaling in Financial Agent Societiescited by 0
  3. 2026Market Signal Injection: Adversarial Context Manipulation of LLM Pricing Agentscited by 0
  4. 2026Xeno-Interpretability: Investigating the Alien Minds of LLMscited by 0
  5. 2026A Society of Researchers: Designing Institutions for Populations of Autonomous Research Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic AI, Retrieval-Augmented Generation, and the Institutional Turn: Legal Architectures and Financial Governance in the Age of Distributional AGI
  2. 2026Alignment as Institutional Design: From Behavioral Correction to Transaction Structure in Intelligent Systems
  3. 2026Compliant with Local Controls, Collectively Discriminatory. A Governance Architecture for Multi-Agent AI in Regulated Finance
  4. 2026Artificial Institutions: How Institutional Design Shapes LLM Simulations
  5. 2026I Can't Believe It's Corrupt: Evaluating Corruption in Multi-Agent Governance Systems
  6. 2026On the Fragility of AI Agent Collusion
  7. 2026Voluntary Collusion with Secret Tools in Competing LLM Agents
  8. 2025Governance-as-a-Service: A Multi-Agent Framework for AI System Compliance and Policy Enforcement