# Cooperative and uncooperative institution designs: Surprises and problems in open-source game theory

arXiv: https://arxiv.org/abs/2208.07006 · HTML: https://mlsanity.com/paper/2208.07006/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2022.

## Built on by (most cited first, 11 of 11)

- [Cooperative AI via Decentralized Commitment Devices](https://mlsanity.com/paper/2311.07815/) (2023)
- [Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value](https://mlsanity.com/paper/2512.03399/) (2025)
- [Evaluating LLMs in Open-Source Games](https://mlsanity.com/paper/2512.00371/) (2025)
- [Similarity-based cooperative equilibrium](https://mlsanity.com/paper/2211.14468/) (2022)
- [Recursive Joint Simulation in Games](https://mlsanity.com/paper/2402.08128/) (2024)
- [Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI](https://mlsanity.com/paper/2605.08426/) (2026)
- [Red Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents](https://mlsanity.com/paper/2209.02167/) (2022)
- [Policy-Conditioned Policies for Multi-Agent Task Solving](https://mlsanity.com/paper/2512.21024/) (2025)
- [Causal Foundations of Collective Agency](https://mlsanity.com/paper/2605.00248/) (2026)
- [Parametric Open Source Games](https://mlsanity.com/paper/2606.27068/) (2026)
- [Math for AI safety: an invitation for mathematicians](https://mlsanity.com/paper/2609.15289/) (2026)

## Abstract (arXiv)

It is increasingly possible for real-world agents, such as software-based agents or human institutions, to view the internal programming of other such agents that they interact with. For instance, a company can read the bylaws of another company, or one software system can read the source code of another. Game-theoretic equilibria between the designers of such agents are called program equilibria, and we call this area open-source game theory. In this work we demonstrate a series of counterintuitive results on open-source games, which are independent of the programming language in which agents are written. We show that certain formal institution designs that one might expect to defect against each other will instead turn out to cooperate, or conversely, cooperate when one might expect them to defect. The results hold in a setting where each institution has full visibility into the other institution's true operating procedures. We also exhibit examples and ten open problems for better understanding these phenomena. We argue that contemporary game theory remains ill-equipped to study program equilibria, given that even the outcomes of single games in open-source settings remain counterintuitive and poorly understood. Nonetheless, some of these open-source agents exhibit desirable characteristics -- e.g., they can unexploitably create incentives for cooperation and legibility from other agents -- such that analyzing them could yield considerable benefits.