mlsanityData to 7 Oct 2026

Efficacy of Language Model Self-Play in Non-Zero-Sum Games

arXiv 2406.18872 ↗First posted 27 Jun 2024cs.CLcited by 14

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning

Abstract (arXiv)

Game-playing agents like AlphaGo have achieved superhuman performance through self-play, which is theoretically guaranteed to yield optimal policies in competitive games. However, most language tasks are partially or fully cooperative, so it is an open question whether techniques like self-play can effectively be used to improve language models. We empirically investigate this question in a negotiation game setting known as Deal or No Deal (DoND). Crucially, the objective in DoND can be modified to produce a fully cooperative game, a strictly competitive one, or anything in between. We finetune language models in self-play over multiple rounds of filtered behavior cloning in DoND for each of these objectives and evaluate them in self-play and in collaboration with humans. We find that language models improve substantially in self-play, achieving 14-17x higher scores in task reward after finetuning. Further, the trained models generalize to both cooperation and competition with humans, scoring 2.5-6x higher than base models. We view these results as an early promising sign for language model self-play in cooperative settings, despite a lack of theoretical guarantees.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 40 of its 55 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

40 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback17 May

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  10. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  11. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  12. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
27 more
  1. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  2. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  3. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 156
  4. 2021Collaborating with Humans without Human Datacited by 139
  5. 2023Vision-Language Models as Success Detectorscited by 138
  6. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  7. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  8. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 112
  9. 2020Near-Optimal Reinforcement Learning with Self-Playcited by 108
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  11. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  12. 2023MindAgent: Emergent Gaming Interactioncited by 87
  13. 2017Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialogcited by 86
  14. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  15. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  16. 2023Strategic Reasoning with Language Modelscited by 64
  17. 2021CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systemscited by 60
  18. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 52
  19. 2019A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Contextcited by 35
  20. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  21. 2023Decision-Oriented Dialogue for Human-AI Collaborationcited by 29
  22. 2024Steering Language Models with Game-Theoretic Solverscited by 28
  23. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  24. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  25. 2021Reference-Centric Models for Grounded Collaborative Dialoguecited by 15
  26. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  27. 2024Policy Learning with a Language Bottleneckcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  2. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  3. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  4. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  5. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  6. 2025Towards Strategic Persuasion with Language Modelscited by 7
  7. 2025Learning Robust Social Strategies with Large Language Modelscited by 2
  8. 2026Interactive Critique-Revision Training for Reliable Structured LLM Generationcited by 2
  9. 2026Talk is Cheap, Communication is Hard: Dynamic Grounding Failures and Repair in Multi-Agent Negotiationcited by 1
  10. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 1
  11. 2026The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Playcited by 0
  12. 2026Counterparty Modeling is Not Strategy: The Limits of LLM Negotiatorscited by 0
2 more
  1. 2026INFUSER: Influence-Guided Self-Evolution Improves Reasoningcited by 0
  2. 2026From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Collaboration and Conflict between Humans and Language Models through the Lens of Game Theory
  2. 2025Language Self-Play For Data-Free Training
  3. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agents
  4. 2026Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Games
  5. 2025LLMs as Agentic Cooperative Players in Multiplayer UNO
  6. 2024Can Large Language Models Play Games? A Case Study of A Self-Play Approach
  7. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
  8. 2025Playing games with Large language models: Randomness and strategy