# Cultural Evolution of Cooperation among LLM Agents

arXiv: https://arxiv.org/abs/2412.10270 · HTML: https://mlsanity.com/paper/2412.10270/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [The Ethics of Advanced AI Assistants](https://mlsanity.com/paper/2404.16244/) (2024)
- [CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agents](https://mlsanity.com/paper/2310.17512/) (2023)
- [Strategic Reasoning with Language Models](https://mlsanity.com/paper/2305.19165/) (2023)
- [Machine Culture](https://mlsanity.com/paper/2311.11388/) (2023)
- [Do LLM Agents Exhibit Social Behavior?](https://mlsanity.com/paper/2312.15198/) (2023)
- [Artificial Leviathan: Exploring Social Evolution of LLM Agents Through the Lens of Hobbesian Social Contract Theory](https://mlsanity.com/paper/2406.14373/) (2024)
- [Social Learning: Towards Collaborative Learning with Large Language Models](https://mlsanity.com/paper/2312.11441/) (2023)
- [Cultural evolution in populations of Large Language Models](https://mlsanity.com/paper/2403.08882/) (2024)
- [Collective Innovation in Groups of Large Language Models](https://mlsanity.com/paper/2407.05377/) (2024)

## Built on by (most cited first, 30 of 32)

- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games](https://mlsanity.com/paper/2506.23276/) (2025)
- [Spontaneous Giving and Calculated Greed in Language Models](https://mlsanity.com/paper/2502.17720/) (2025)
- [How Personality Traits Shape LLM Risk-Taking Behaviour](https://mlsanity.com/paper/2503.04735/) (2025)
- [The Role of Social Learning and Collective Norm Formation in Fostering Cooperation in LLM Multi-Agent Systems](https://mlsanity.com/paper/2510.14401/) (2025)
- [AstroAgents: A Multi-Agent AI for Hypothesis Generation from Mass Spectrometry Data](https://mlsanity.com/paper/2503.23170/) (2025)
- [Modeling Earth-Scale Human-Like Societies with One Billion Agents](https://mlsanity.com/paper/2506.12078/) (2025)
- [Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences](https://mlsanity.com/paper/2510.06105/) (2025)
- [Evaluating Collective Behaviour of Hundreds of LLM Agents](https://mlsanity.com/paper/2602.16662/) (2026)
- [OpenClaw AI Agents as Informal Learners at Moltbook: Characterizing an Emergent Learning Community at Scale](https://mlsanity.com/paper/2602.18832/) (2026)
- [More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration](https://mlsanity.com/paper/2604.07821/) (2026)
- [Emergent Coordinated Behaviors in Networked LLM Agents: Modeling the Strategic Dynamics of Information Operations](https://mlsanity.com/paper/2510.25003/) (2025)
- [Gender Dynamics and Homophily in a Social Network of LLM Agents](https://mlsanity.com/paper/2602.02606/) (2026)
- [Talk, Judge, Cooperate: Gossip-Driven Indirect Reciprocity in Self-Interested LLM Agents](https://mlsanity.com/paper/2602.07777/) (2026)
- [When AI Agents Teach Each Other: Discourse Patterns Resembling Peer Learning in the Moltbook Community](https://mlsanity.com/paper/2602.14477/) (2026)
- [Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?](https://mlsanity.com/paper/2504.03814/) (2025)
- [From Connectivity to Autonomy: The Dawn of Self-Evolving Communication Systems](https://mlsanity.com/paper/2505.23710/) (2025)
- [Learning to Learn from Language Feedback with Social Meta-Learning](https://mlsanity.com/paper/2602.16488/) (2026)
- [When AI Agents Learn from Each Other: Insights from Emergent AI Agent Communities on OpenClaw for Human-AI Partnership in Education](https://mlsanity.com/paper/2603.16663/) (2026)
- [Emergence of Reputation-Based Cooperation in LLM Agents](https://mlsanity.com/paper/2608.04507/) (2026)
- [Behavior is Not Enough: A Mechanism-Based Evaluation of Social Norm Emergence in LLM Societies](https://mlsanity.com/paper/2609.26481/) (2026)
- [Position: Introspective Experience from Conversational Environments as a Path to Better Learning](https://mlsanity.com/paper/2602.14910/) (2026)
- [Next-Token Prediction and Regret Minimization](https://mlsanity.com/paper/2603.28499/) (2026)
- [You Only Align Once: Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents](https://mlsanity.com/paper/2605.27586/) (2026)
- [Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension](https://mlsanity.com/paper/2605.29874/) (2026)
- [Emergent Relational Order in LLM Agent Societies: From Collective Affect to Authority Stratification](https://mlsanity.com/paper/2606.23764/) (2026)
- [Not a Monolith: Lab-Level Divergence in the Cooperative Equilibria of Chinese Frontier LLM Agents](https://mlsanity.com/paper/2608.10262/) (2026)
- [Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generation](https://mlsanity.com/paper/2609.04484/) (2026)
- [Playing social deduction games with reinforcement fine-tuned large language models](https://mlsanity.com/paper/2610.04261/) (2026)
- [Communication Shapes Collective Inference in Self-Adapting LLM Societies: Evidence from Mafia](https://mlsanity.com/paper/2610.05041/) (2026)

## Abstract (arXiv)

Large language models (LLMs) provide a compelling foundation for building generally-capable AI agents. These agents may soon be deployed at scale in the real world, representing the interests of individual humans (e.g., AI assistants) or groups of humans (e.g., AI-accelerated corporations). At present, relatively little is known about the dynamics of multiple LLM agents interacting over many generations of iterative deployment. In this paper, we examine whether a "society" of LLM agents can learn mutually beneficial social norms in the face of incentives to defect, a distinctive feature of human sociality that is arguably crucial to the success of civilization. In particular, we study the evolution of indirect reciprocity across generations of LLM agents playing a classic iterated Donor Game in which agents can observe the recent behavior of their peers. We find that the evolution of cooperation differs markedly across base models, with societies of Claude 3.5 Sonnet agents achieving significantly higher average scores than Gemini 1.5 Flash, which, in turn, outperforms GPT-4o. Further, Claude 3.5 Sonnet can make use of an additional mechanism for costly punishment to achieve yet higher scores, while Gemini 1.5 Flash and GPT-4o fail to do so. For each model class, we also observe variation in emergent behavior across random seeds, suggesting an understudied sensitive dependence on initial conditions. We suggest that our evaluation regime could inspire an inexpensive and informative new class of LLM benchmarks, focussed on the implications of LLM agent deployment for the cooperative infrastructure of society.