# Causal Foundations of Collective Agency

arXiv: https://arxiv.org/abs/2605.00248 · HTML: https://mlsanity.com/paper/2605.00248/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 70 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 30 of 33)

- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents](https://mlsanity.com/paper/2404.16698/) (2024)
- [Causal Consistency of Structural Equation Models](https://mlsanity.com/paper/1707.00819/) (2017)
- [Abstracting Causal Models](https://mlsanity.com/paper/1812.03789/) (2018)
- [Approximate Causal Abstraction](https://mlsanity.com/paper/1906.11583/) (2019)
- [Agent Incentives: A Causal Perspective](https://mlsanity.com/paper/2102.01685/) (2021)
- [Compositional Abstraction Error and a Category of Causal Models](https://mlsanity.com/paper/2103.15758/) (2021)
- [Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility](https://mlsanity.com/paper/1810.05903/) (2018)
- [Adversaries Can Misuse Combinations of Safe Models](https://mlsanity.com/paper/2406.14595/) (2024)
- [Discovering Agents](https://mlsanity.com/paper/2208.08345/) (2022)
- [Causal Abstraction with Soft Interventions](https://mlsanity.com/paper/2211.12270/) (2022)
- [Designing Ecosystems of Intelligence from First Principles](https://mlsanity.com/paper/2212.01354/) (2022)
- [The Reasons that Agents Act: Intention and Instrumental Goals](https://mlsanity.com/paper/2402.07221/) (2024)
- [Neural Causal Abstractions](https://mlsanity.com/paper/2401.02602/) (2024)
- [Reasoning about Causality in Games](https://mlsanity.com/paper/2301.02324/) (2023)
- [Agents and Devices: A Relative Definition of Agency](https://mlsanity.com/paper/1805.12387/) (2018)
- [Evaluating the Goal-Directedness of Large Language Models](https://mlsanity.com/paper/2504.11844/) (2025)
- [Empirical Game-Theoretic Analysis: A Survey](https://mlsanity.com/paper/2403.04018/) (2024)
- [Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory](https://mlsanity.com/paper/2208.07006/) (2022)
- [Towards Measuring Goal-Directedness in AI Systems](https://mlsanity.com/paper/2410.04683/) (2024)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [What is causal about causal models and representations?](https://mlsanity.com/paper/2501.19335/) (2025)
- [Agency Is Frame-Dependent](https://mlsanity.com/paper/2502.04403/) (2025)
- [Similarity-based cooperative equilibrium](https://mlsanity.com/paper/2211.14468/) (2022)
- [Partial Identifiability and Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15951/) (2024)
- [When and Why Does Multi-Agent Debate Fail and Does It Really Underperform?](https://mlsanity.com/paper/2510.20963/) (2025)
- [The Case for Time in Causal DAGs](https://mlsanity.com/paper/2501.19311/) (2025)

## Abstract (arXiv)

A key challenge for the safety of advanced AI systems is the possibility that multiple simpler agents might inadvertently form a collective agent with capabilities and goals distinct from those of any individual. More generally, determining when a group of agents can be viewed as a unified collective agent is a foundational question in the study of interactions and incentives in both biological and artificial systems. We adopt a behavioral perspective in answering this question, ascribing collective agency to a group when viewing the group's joint actions as rational and goal-directed successfully predicts its behavior. We formalize this perspective on collective agency using causal games -- which are causal models of strategic, multi-agent interactions -- and causal abstraction -- which formalizes when a simple, high-level model faithfully captures a more complex, low-level model. We use this framework to solve a puzzle regarding multi-agent incentives in actor-critic models and to make quantitative assessments of the degree of collective agency exhibited by different voting mechanisms. Our framework aims to provide a foundation for theoretical and empirical work to understand, predict, and control emergent collective agents in multi-agent AI systems.