# CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

arXiv: https://arxiv.org/abs/2604.15267 · HTML: https://mlsanity.com/paper/2604.15267/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 14 of its 135 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 14 of 14)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmas](https://mlsanity.com/paper/2505.19212/) (2025)
- [A multi-agent reinforcement learning model of reputation and cooperation in human groups](https://mlsanity.com/paper/2103.04982/) (2021)
- [GT-HarmBench: Benchmarking AI Safety Risks Through the Lens of Game Theory](https://mlsanity.com/paper/2602.12316/) (2026)
- [Recursive Joint Simulation in Games](https://mlsanity.com/paper/2402.08128/) (2024)
- [Strategies of cooperation and defection in five large language models](https://mlsanity.com/paper/2601.09849/) (2026)
- [Improving Social Welfare While Preserving Autonomy via a Pareto Mediator](https://mlsanity.com/paper/2106.03927/) (2021)
- [How large language models judge and influence human cooperation](https://mlsanity.com/paper/2507.00088/) (2025)
- [Maximizing Social Welfare with Side Payments](https://mlsanity.com/paper/2508.07147/) (2025)

## Built on by (most cited first, 16 of 16)

- [The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agents](https://mlsanity.com/paper/2605.08060/) (2026)
- [Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI](https://mlsanity.com/paper/2605.08426/) (2026)
- [CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experiments](https://mlsanity.com/paper/2606.06399/) (2026)
- [Incoherent Values? Probing LLM Preferences Through Parametric Variation](https://mlsanity.com/paper/2606.21102/) (2026)
- [Cooperative Profiles Predict Multi-Agent LLM Team Performance in AI for Science Workflows](https://mlsanity.com/paper/2604.20658/) (2026)
- [Draining the Energy Commons: Self-Defeating Over-Appropriation as a Coordination Failure in Agentic LLM Collectives](https://mlsanity.com/paper/2607.22188/) (2026)
- [Group Selection Promotes Prosocial Prompts in Populations of LLM Agents](https://mlsanity.com/paper/2606.23343/) (2026)
- [When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games](https://mlsanity.com/paper/2607.05132/) (2026)
- [Formal Mechanisms for Market Stability in Self-Interested Agent Societies: A Marketplace Simulation Study](https://mlsanity.com/paper/2607.08652/) (2026)
- [Moral Hazard in Multi-Agent Language Models](https://mlsanity.com/paper/2607.23982/) (2026)
- [Evaluating Rational Contracting in Natural Language](https://mlsanity.com/paper/2608.10475/) (2026)
- [The Open-Strategy Dictator Game: Cooperation Under Mutual Transparency](https://mlsanity.com/paper/2608.14913/) (2026)
- [From Certain Doom to Survival: Agent-Driven Self-Governance in LLM Agent Societies](https://mlsanity.com/paper/2609.22600/) (2026)
- [Financial Fragility in Societies of LLM Agents: Coordination Failures and Stabilizing Mechanisms](https://mlsanity.com/paper/2609.30940/) (2026)
- [Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams](https://mlsanity.com/paper/2610.00583/) (2026)
- [Identity, Incentives, and Cooperative Behavior for AI: Reverse-Engineering Multiagent Cooperation](https://mlsanity.com/paper/2610.19188/) (2026)

## Abstract (arXiv)

It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave _less_ cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings. Indeed, our experiments show that recent models -- with or without reasoning enabled -- consistently defect in single-shot social dilemmas. To tackle this safety concern, we present the first comparative study of game-theoretic mechanisms designed to enable cooperative outcomes between rational agents _in equilibrium_. Across four social dilemmas testing distinct components of robust cooperation, we evaluate four families of mechanisms: (1) repeating the game for many rounds, (2) reputation systems, (3) third-party mediators to delegate decision making to, and (4) contract agreements for outcome-conditional payments between players. Among our findings, we establish that contracting and mediation are most effective in achieving cooperative outcomes between capable LLM models, and that repetition-induced cooperation deteriorates drastically when co-players vary. Moreover, we demonstrate that the mechanisms become _more effective_ under evolutionary pressures to maximize individual payoffs.