Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
We introduce institutional red-teaming, an evaluation methodology for testing deployment rules in multi-agent AI: hold the agents, objectives, and task state fixed, vary only one rule, and attribute the resulting change in collective behavior to that rule. We instantiate the methodology in IABench-CA, a consequence-allocation benchmark spanning 228 contexts, five canonical rules, and seven model populations (33,924 games), with a normative cooperative reference and auto-labelled reasoning traces. Three findings emerge. (1) Deployment rules causally alter collective safety: changing only the consequence rule moves mean fatality by 22 to 58 percentage points within every population. (2) There is no safe default, but the targeting hazard is universal: the safest rule, the least-safe rule, and even the direction of the incidence effect vary across populations, yet regressive identity-targeting is never decisively safest in any context for any population, eliminates the least-resourced agent in 30-87% of games everywhere, and is selection-unsafe relative to the cooperative reference for all seven populations. (3) Identity salience is the mechanism: a one-shot anonymization ablation on the most exploitation-prone population (gpt-5.1) shows that merely naming the loss bearer in the rule text drives targeted elimination from 22% to 81% at identical payoffs; under repeated play, anonymization only delays the targeting, as agents re-infer the hidden rule from observed eliminations. We package the methodology as a safety-case workflow that certifies a provisional rule region Phi(c,P) per deployment context and population, with explicit residual risks and monitoring obligations.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 12 of its 23 references that are in the map. First posted on arXiv 8 Jul 2026.
Builds on
12 of 23 references.
References · 12
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2016Concrete Problems in AI Safetycited by 1,642
- 2018AI safety via debatecited by 400
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
- 2020Open Problems in Cooperative AIcited by 168
- 2025Multi-Agent Risks from Advanced AIcited by 163
- 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
- 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
- 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026Multi-Agent AI Safety as an Institutional Design Problemcited by 1
- 2026Multi-Agent AI Safety as an Institutional Design Problem10 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols
- 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition
- 2025Red Teaming AI Red Teaming
- 2026Institutional AI: A Governance Framework for Distributional AGI Safety
- 2026Trust Between AI Agents: Measuring Formation, Breakage, and Recovery, with Implications for Governing Multi-Agent Systems
- 2026Position: Safety and Fairness in Agentic AI Depend on Interaction Topology, Not on Model Scale or Alignment
- 2025The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy
- 2026Agentic Microphysics: A Manifesto for Generative AI Safety