mlsanityData to 7 Oct 2026

Self-playing Adversarial Language Game Enhances LLM Reasoning

arXiv 2404.10642 ↗First posted 16 Apr 2024cs.CL cs.LGcited by 80

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

We explore the potential of self-play training for large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players must have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Playing this Adversarial language Game (SPAG). With this goal, we select several open-source LLMs and let each act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is available at https://github.com/Linear95/SPAG.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 49 of its 73 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

49 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Enhance Reasoning for Large Language Models in the Game Werewolf4 Feb

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2015Trust Region Policy Optimizationcited by 2,503
36 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  5. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  8. 2022Large Language Models Can Self-Improvecited by 656
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  13. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  14. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  15. 2023Llemma: An Open Language Model For Mathematicscited by 385
  16. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  17. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  18. 2023How Language Model Hallucinations Can Snowballcited by 310
  19. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  20. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  21. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  22. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  23. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  24. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  25. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  26. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  27. 2018Self-Imitation Learningcited by 135
  28. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  29. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 114
  30. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  31. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  32. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  33. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  34. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  35. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  36. 2019Adversarial Language Games for Advanced Natural Language Intelligencecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 43; 2026: 25.

80 papers.

Sort:

By area:

  1. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 323
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  4. 2025Learning to Reason without External Rewardscited by 199
  5. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  6. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  7. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  8. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  9. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  10. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
  11. 2025Dynamic Parallel Tree Search for Efficient LLM Reasoningcited by 42
  12. 2025Language Self-Play For Data-Free Trainingcited by 42
38 more
  1. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  2. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  3. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  4. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  5. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  6. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  7. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  8. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 31
  9. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  10. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 18
  11. 2024Strategist: Self-improvement of LLM Decision Making via Bi-Level Tree Searchcited by 16
  12. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  13. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safetycited by 14
  14. 2024Toward Optimal LLM Alignments Using Two-Player Gamescited by 13
  15. 2025Lifelong Safety Alignment for Language Modelscited by 13
  16. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 11
  17. 2025Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Gamecited by 10
  18. 2025Anyprefer: An Agentic Framework for Preference Data Synthesiscited by 10
  19. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  20. 2025Evaluation and Facilitation of Online Discussions in the LLM Era: A Surveycited by 8
  21. 2025Incentivizing Truthful Language Models via Peer Elicitation Gamescited by 8
  22. 2026Learning Robust Reasoning through Guided Adversarial Self-Playcited by 8
  23. 2024Atoxia: Red-teaming Large Language Models with Target Toxic Answerscited by 7
  24. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Gamecited by 7
  25. 2025What Can You Do When You Have Zero Rewards During RL?cited by 7
  26. 2025Better LLM Reasoning via Dual-Playcited by 7
  27. 2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7
  28. 2024Direct Alignment of Language Models via Quality-Aware Self-Refinementcited by 6
  29. 2025CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Gamescited by 6
  30. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  31. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 6
  32. 2025AI Agent Behavioral Sciencecited by 5
  33. 2025SPFT-SQL: Enhancing Large Language Model for Text-to-SQL Parsing by Self-Play Fine-Tuningcited by 5
  34. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 5
  35. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  36. 2024Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learningcited by 4
  37. 2025Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solvingcited by 4
  38. 2025clem:todd: A Framework for the Systematic Benchmarking of LLM-Based Task-Oriented Dialogue System Realisationscited by 4

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
  2. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
  3. 2025Towards Understanding Self-play for LLM Reasoning
  4. 2026Seirenes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
  5. 2025Adversarial Testing in LLMs: Insights into Decision-Making Vulnerabilities
  6. 2025Scheming Ability in LLM-to-LLM Strategic Interactions
  7. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  8. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data