# Evolution of Cooperation in LLM-Agent Societies: A Preliminary Study Using Different Punishment Strategies

arXiv: https://arxiv.org/abs/2504.19487 · HTML: https://mlsanity.com/paper/2504.19487/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 19 of 19)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [ScienceWorld: Is your Agent Smarter than a 5th Grader?](https://mlsanity.com/paper/2203.07540/) (2022)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis](https://mlsanity.com/paper/2312.05488/) (2023)
- [Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?](https://mlsanity.com/paper/2406.13605/) (2024)
- [Do LLM Agents Exhibit Social Behavior?](https://mlsanity.com/paper/2312.15198/) (2023)
- [Can Language Models Serve as Text-Based World Simulators?](https://mlsanity.com/paper/2406.06485/) (2024)
- [Emergence of Social Norms in Generative Agent Societies: Principles and Architecture](https://mlsanity.com/paper/2403.08251/) (2024)
- [Melting Pot 2.0](https://mlsanity.com/paper/2211.13746/) (2022)
- [ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agents](https://mlsanity.com/paper/2311.03220/) (2023)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Harnessing the power of LLMs for normative reasoning in MASs](https://mlsanity.com/paper/2403.16524/) (2024)
- [Norm Violation Detection in Multi-Agent Systems using Large Language Models: A Pilot Study](https://mlsanity.com/paper/2403.16517/) (2024)
- [Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges](https://mlsanity.com/paper/2404.02269/) (2024)
- [Cooperative bots exhibit nuanced effects on cooperation across strategic frameworks](https://mlsanity.com/paper/2406.14913/) (2024)

## Built on by (most cited first, 6 of 6)

- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [The Role of Social Learning and Collective Norm Formation in Fostering Cooperation in LLM Multi-Agent Systems](https://mlsanity.com/paper/2510.14401/) (2025)
- [Gender Dynamics and Homophily in a Social Network of LLM Agents](https://mlsanity.com/paper/2602.02606/) (2026)
- [ScheduleMe: Multi-Agent Calendar Assistant](https://mlsanity.com/paper/2509.25693/) (2025)
- [Social Norm Reasoning in Multimodal Language Models: An Evaluation](https://mlsanity.com/paper/2603.03590/) (2026)
- [Behavior is Not Enough: A Mechanism-Based Evaluation of Social Norm Emergence in LLM Societies](https://mlsanity.com/paper/2609.26481/) (2026)

## Abstract (arXiv)

The evolution of cooperation has been extensively studied using abstract mathematical models and simulations. Recent advances in Large Language Models (LLMs) and the rise of LLM agents have demonstrated their ability to perform social reasoning, thus providing an opportunity to test the emergence of norms in more realistic agent-based simulations with human-like reasoning using natural language. In this research, we investigate whether the cooperation dynamics presented in Boyd and Richerson's model persist in a more realistic simulation of the Diner's Dilemma using LLM agents compared to the abstract mathematical nature in the work of Boyd and Richerson. Our findings indicate that agents follow the strategies defined in the Boyd and Richerson model, and explicit punishment mechanisms drive norm emergence, reinforcing cooperative behaviour even when the agent strategy configuration varies. Our results suggest that LLM-based Multi-Agent System simulations, in fact, can replicate the evolution of cooperation predicted by the traditional mathematical models. Moreover, our simulations extend beyond the mathematical models by integrating natural language-driven reasoning and a pairwise imitation method for strategy adoption, making them a more realistic testbed for cooperative behaviour in MASs.