# Emergent Collusion in Long-Horizon LLM Agent Interaction

arXiv: https://arxiv.org/abs/2609.24967 · HTML: https://mlsanity.com/paper/2609.24967/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 28 of 28)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems](https://mlsanity.com/paper/2410.07283/) (2024)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities](https://mlsanity.com/paper/2407.07791/) (2024)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [AgentRxiv: Towards Collaborative Autonomous Research](https://mlsanity.com/paper/2503.18102/) (2025)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)
- [ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases](https://mlsanity.com/paper/2510.20270/) (2025)
- [Algorithmic Collusion by Large Language Models](https://mlsanity.com/paper/2404.00806/) (2024)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [Monitoring Monitorability](https://mlsanity.com/paper/2512.18311/) (2025)
- [Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs](https://mlsanity.com/paper/2410.03768/) (2024)
- [Do as We Do, Not as You Think: the Conformity of Large Language Models](https://mlsanity.com/paper/2501.13381/) (2025)
- [CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://mlsanity.com/paper/2601.13295/) (2026)
- [Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2602.15198/) (2026)
- [When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systems](https://mlsanity.com/paper/2507.14660/) (2025)
- [When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms](https://mlsanity.com/paper/2511.06448/) (2025)
- [Detecting Multi-Agent Collusion Through Multi-Agent Interpretability](https://mlsanity.com/paper/2604.01151/) (2026)
- [Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agents](https://mlsanity.com/paper/2604.17774/) (2026)
- [Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds](https://mlsanity.com/paper/2606.15385/) (2026)
- [Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisions](https://mlsanity.com/paper/2608.18078/) (2026)

## Built on by (most cited first, 2 of 2)

- [Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident](https://mlsanity.com/paper/2610.00902/) (2026)
- [SwarmReconGuard: Black-Box Detection of Distributed Collective Reconnaissance by Individually Benign-Looking Agent Populations](https://mlsanity.com/paper/2610.09138/) (2026)

## Abstract (arXiv)

LLM agents are increasingly deployed in collaborative settings, yet long-term interaction may give rise to undesirable coordination. We study the emergence of collusion in a long-horizon multi-agent environment: two agents repeatedly complete individual tasks, share task logs, verify each other's work, and receive rewards. We introduce realistic constraints under which higher reward is attainable only by violating the verification protocol, and find that agents increasingly deviate from the protocol over repeated interactions. Collusion emerges in 94% of trajectories across 10 models, and more capable models within the same family reach it earlier. Controlled peer interventions show that collusion is shaped by peer behavior, while ablations reveal additional effects of the verification feedback agents receive, their interaction history, and the reward structure. In particular, restricting the amount and scope of interaction history available to agents reduces collusion. Overall, our findings show that long-horizon interaction can reshape how agents coordinate in ways that create safety risks.