# Multi-Agent AI Safety as an Institutional Design Problem

arXiv: https://arxiv.org/abs/2608.09828 · HTML: https://mlsanity.com/paper/2608.09828/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 24 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework](https://mlsanity.com/paper/2601.11893/) (2026)
- [Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs](https://mlsanity.com/paper/2601.11369/) (2026)
- [ChainCaps: Composition-Safe Tool-Using Agents via Monotonic Capability Attenuation](https://mlsanity.com/paper/2605.26542/) (2026)
- [Authorization Propagation in Multi-Agent AI Systems: Identity Governance as Infrastructure](https://mlsanity.com/paper/2605.05440/) (2026)
- [Safeguarding LLM Agents from Misalignment through Provenance Analysis](https://mlsanity.com/paper/2607.01236/) (2026)
- [Normative Modules: A Generative Agent Architecture for Learning Norms that Supports Multi-Agent Cooperation](https://mlsanity.com/paper/2405.19328/) (2024)
- [Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI](https://mlsanity.com/paper/2605.08426/) (2026)
- [Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safety](https://mlsanity.com/paper/2607.07097/) (2026)
- [Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety](https://mlsanity.com/paper/2607.07695/) (2026)
- [APPA: Recoverable Information-Flow Control for Real-World LLM Agents](https://mlsanity.com/paper/2607.24625/) (2026)

## Built on by (most cited first, 1 of 1)

- [Where Reliability Lives: Experimental Localisation of Behavioural Properties in an Agent System](https://mlsanity.com/paper/2609.03192/) (2026)

## Abstract (arXiv)

AI agents increasingly work inside systems that govern how they delegate tasks, move information, execute actions, and use shared resources. Recent work already shows that deployment rules can change collective behavior. Here we ask which parts of an AI institution produce safety and how they do it. This is the first paper from POLIS, an ongoing research programme studying algorithmic institutions for multi-agent systems. We report a frozen 5,280-episode study suite. The main pre-specified delegation experiment spans four model families; a targeted high-conflict diagnostic adds three additional model endpoints. In matched structured workflows, the model sees different rule formulations and guards consult different authority states. We also vary the attractiveness of the immediate compliant internal/self fallback and allow blocked workflows to continue. A detailed constitutional prompt produces 0/384 realized violations. A provenance-aware executable guard also produces 0/384, although it blocks prohibited attempts in 51/384 episodes; 44/51 of those episodes later complete safely. The local-state guard's failures concentrate in scenarios where an ordinary transformation changes visible policy while originating authority stays fixed. In matched laundering scenarios, that guard admits violations in 22/96 episodes and provenance enforcement in 0/96 (p = 4.77 x 10^-7). A separate resource-allocation experiment shows that revealing the numerical value of an otherwise identical cap changes agent requests. In these structured workflows, the same final violation rate can hide very different mechanisms. The rule itself is only part of the institution. The authority state the system trusts matters, and so does the path available after a block.