# Multi-Agent Systems Should be Treated as Principal-Agent Problems

arXiv: https://arxiv.org/abs/2601.23211 · HTML: https://mlsanity.com/paper/2601.23211/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 37 of its 60 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 30 of 37)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Large Language Models can Strategically Deceive their Users when Put Under Pressure](https://mlsanity.com/paper/2311.07590/) (2023)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [Recursive Language Models](https://mlsanity.com/paper/2512.24601/) (2025)
- [Deception Abilities Emerged in Large Language Models](https://mlsanity.com/paper/2307.16513/) (2023)
- [Scheming AIs: Will AIs fake alignment during training in order to get power?](https://mlsanity.com/paper/2311.08379/) (2023)
- [Stress Testing Deliberative Alignment for Anti-Scheming Training](https://mlsanity.com/paper/2509.15541/) (2025)
- [Truth is Universal: Robust Detection of Lies in LLMs](https://mlsanity.com/paper/2407.12831/) (2024)
- [Incomplete Contracting and AI Alignment](https://mlsanity.com/paper/1804.04268/) (2018)
- [Agent Modelling under Partial Observability for Deep Reinforcement Learning](https://mlsanity.com/paper/2006.09447/) (2020)
- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [LLMs Can Generate a Better Answer by Aggregating Their Own Responses](https://mlsanity.com/paper/2503.04104/) (2025)
- [AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents](https://mlsanity.com/paper/2506.04018/) (2025)
- [The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process](https://mlsanity.com/paper/2506.01080/) (2025)

## Built on by (most cited first, 3 of 3)

- [MADE: Belief-Driven Dual-Agent Coordination for Autonomous Model Deployment](https://mlsanity.com/paper/2608.01189/) (2026)
- [When Intelligence Becomes Agency: A Theory of Governed, Proactive Agency for Symbiotic AI Systems](https://mlsanity.com/paper/2609.07741/) (2026)
- [From Token-Max to Outcome-Max: How You Use AI Determines Its Productivity](https://mlsanity.com/paper/2610.05697/) (2026)

## Abstract (arXiv)

Consider a multi-agent systems setup in which a principal (a supervisor agent) assigns subtasks to specialized agents and aggregates their responses into a single system-level output. A core property of such systems is information asymmetry: agents observe task-specific information, produce intermediate reasoning traces, and operate with different context windows. In isolation, such asymmetry is not problematic, since agents report truthfully to the principal when incentives are fully aligned. However, this assumption breaks down when incentives diverge. Recent evidence suggests that LLM-based agents can acquire their own goals, such as survival or self-preservation, a phenomenon known as scheming, and may deceive humans or other agents. This leads to agency loss: a gap between the principal's intended outcome and the realized system behavior. Drawing on core ideas from microeconomic theory, we argue that these characteristics, information asymmetry and misaligned goals, are best studied through the lens of principal-agent problems. We explain why multi-agent systems, both human-to-LLM and LLM-to-LLM, naturally induce information asymmetry under this formulation, and we use scheming, where LLM agents pursue covert goals, as a concrete case study. We show that recently introduced terminology used to describe scheming, such as covert subversion or deferred subversion, corresponds to well-studied concepts in the mechanism design literature, which not only characterizes the problem but also prescribes concrete mitigation strategies. More broadly, we argue for applying tools developed to study human agent behavior to the analysis of non-human agents.