mlsanityData to 14 Oct 2026

Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemma

arXiv 2501.16173 ↗First posted 27 Jan 2025cs.MA cs.GTcited by 25

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

As autonomous agents become more prevalent, understanding their collective behaviour in strategic interactions is crucial. This study investigates the emergent cooperative tendencies of systems of Large Language Model (LLM) agents in a social dilemma. Unlike previous research where LLMs output individual actions, we prompt state-of-the-art LLMs to generate complete strategies for iterated Prisoner's Dilemma. Using evolutionary game theory, we simulate populations of agents with different strategic dispositions (aggressive, cooperative, or neutral) and observe their evolutionary dynamics. Our findings reveal that different LLMs exhibit distinct biases affecting the relative success of aggressive versus cooperative strategies. This research provides insights into the potential long-term behaviour of systems of deployed LLM-based autonomous agents and highlights the importance of carefully considering the strategic environments in which they operate.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 24 of its 53 references that are in the map. First posted on arXiv 27 Jan 2025.

Builds on

24 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agents25 Apr
  2. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia6 Dec
  3. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr

Other references · 21

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  8. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  9. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  10. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  11. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 257
  12. 2023Playing repeated games with Large Language Modelscited by 217
9 more
  1. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  2. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
  3. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 98
  4. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  5. 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 92
  6. 2023MindAgent: Emergent Gaming Interactioncited by 88
  7. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  8. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 48
  9. 2024Steering Language Models with Game-Theoretic Solverscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 17.

25 papers.

Sort:

By area:

  1. 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacracited by 22
  2. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  3. 2025When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmascited by 15
  4. 2025FAIRGAME: a Framework for AI Agents Bias Recognition using Game Theorycited by 14
  5. 2025Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agentscited by 12
  6. 2026Evaluating Collective Behaviour of Hundreds of LLM Agentscited by 8
  7. 2026More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaborationcited by 7
  8. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  9. 2025Evaluating LLMs in Open-Source Gamescited by 5
  10. 2025Super-additive Cooperation in Language Model Agentscited by 2
  11. 2026When Numbers Start Talking: Implicit Numerical Coordination Among LLM-Based Agentscited by 2
  12. 2026Communication Enhances LLMs' Stability in Strategic Thinkingcited by 1
13 more
  1. 2026Reasonably reasoning AI agents can avoid game-theoretic failures in zero-shot, provablycited by 1
  2. 2026Conflicts Make Large Reasoning Models Vulnerable to Attackscited by 1
  3. 2026Bosses, Kings, and the Commons: Cooperation Under Power Asymmetry in LLM Societiescited by 1
  4. 2026Used Car Salesbots? Honesty and Credulity of LLMs as Bargaining Agents under Partial Informationcited by 1
  5. 2026Behavior is Not Enough: A Mechanism-Based Evaluation of Social Norm Emergence in LLM Societiescited by 1
  6. 2025Policy-Conditioned Policies for Multi-Agent Task Solvingcited by 0
  7. 2026Learning Incentive Structures for Cooperative Resilience in Multi-Agent Systems under Social Dilemmascited by 0
  8. 2026SODE: Analyzing Social Dynamics in LLM Agentscited by 0
  9. 2026Is Lying an Emergent Behaviour in LLMs? Evidence from Gaslighting AI agents in a Sustainability Gamecited by 0
  10. 2026How Personas Can Influence Agents to Play Split or Stealcited by 0
  11. 2026Cheap Talk Stabilizes Strategic Interaction in LLM Agentscited by 0
  12. 2026When Numbers Start Talking: Numerical Signalling and Strategic Behaviour Among LLMscited by 0
  13. 2026Playing social deduction games with reinforcement fine-tuned large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolution of Cooperation in LLM-Agent Societies: A Preliminary Study Using Different Punishment Strategies
  2. 2026Payoff scaling shapes cooperation in LLM agents across languages
  3. 2026Collective cooperation without individual fidelity in LLM agents
  4. 2024Static network structure cannot stabilize cooperation among Large Language Model agents
  5. 2025Collaboration and Conflict between Humans and Language Models through the Lens of Game Theory
  6. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents
  7. 2026Evolutionary Dynamics of Cooperation in Next-Generation LLM Agent Systems: A Cross-Provider Empirical Extension
  8. 2025Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics