mlsanityData to 14 Oct 2026

Language Self-Play For Data-Free Training

arXiv 2509.07414 ↗First posted 9 Sep 2025cs.AI cs.CL cs.GTcited by 43

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning

Abstract (arXiv)

Large language models (LLMs) have advanced rapidly in recent years, driven by scale, abundant high-quality training data, and reinforcement learning. Yet this progress faces a fundamental bottleneck: the need for ever more data from which models can continue to learn. In this work, we propose a reinforcement learning approach that removes this dependency by enabling models to improve without additional data. Our method leverages a game-theoretic framework of self-play, where a model's capabilities are cast as performance in a competitive game and stronger policies emerge by having the model play against itself-a process we call Language Self-Play (LSP). Experiments with Llama-3.2-3B-Instruct on instruction-following, mathematics, and coding benchmarks show that pretrained models can be effectively improved with self-play alone.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 49 of its 60 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

49 of 60 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  3. 2024Self-playing Adversarial Language Game Enhances LLM Reasoning16 Apr

Other references · 46

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
34 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2016Improved Techniques for Training GANscited by 2,970
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  12. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  13. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  14. 2024Self-Rewarding Language Modelscited by 681
  15. 2022Large Language Models Can Self-Improvecited by 664
  16. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  17. 2019The StarCraft Multi-Agent Challengecited by 597
  18. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  19. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  20. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  21. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  22. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  23. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  24. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 113
  25. 2017The Numerics of GANscited by 105
  26. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  27. 2025Self-Adapting Language Modelscited by 84
  28. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  29. 2025Self-Questioning Language Modelscited by 50
  30. 2018DiCE: The Infinitely Differentiable Monte-Carlo Estimatorcited by 46
  31. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  32. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 41
  33. 2025DataRater: Meta-Learned Dataset Curationcited by 13
  34. 2022Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Gamescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 28.

43 papers.

Sort:

By area:

  1. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  2. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  3. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
  4. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 40
  5. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  6. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  7. 2025A co-evolving agentic AI system for medical imaging analysiscited by 17
  8. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 13
  9. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  10. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  11. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 10
  12. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotationcited by 10
31 more
  1. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Trainingcited by 10
  2. 2026Scaling Self-Play with Self-Guidancecited by 10
  3. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
  4. 2025Better LLM Reasoning via Dual-Playcited by 8
  5. 2026Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesiscited by 8
  6. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 6
  7. 2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 6
  8. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Gamescited by 5
  9. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  10. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  11. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  12. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  13. 2026EvoVid: Temporal-Centric Self-Evolution for Video Large Language Modelscited by 3
  14. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfarecited by 2
  15. 2025Self-Improving AI Agents through Self-Playcited by 2
  16. 2026TTSR: Test-Time Self-Evolving via Reflectioncited by 2
  17. 2026Preventing Curriculum Collapse in Self-Evolving Reasoning Systemscited by 2
  18. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Playcited by 2
  19. 2025Differences That Matter: Auditing Models for Capability Gap Discovery and Rectificationcited by 1
  20. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedbackcited by 1
  21. 2026Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoningcited by 1
  22. 2026Anchored Self-Play for Code Repaircited by 1
  23. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvementcited by 1
  24. 2026J-Zero: Unified Challenger--Solver--Judge Self-Evolution from Zero Datacited by 1
  25. 2025Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novelscited by 0
  26. 2026D2Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learningcited by 0
  27. 2026Bilevel Optimization over Saddle Points of Zero-Sum Markov Gamescited by 0
  28. 2026LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RLcited by 0
  29. 2026Hallucination Self-Play: Bootstrapping Reinforced Detector via Evolved Generatorcited by 0
  30. 2026Ask-E: An Environment for Calibrated Question Generationcited by 0
  31. 2026CompassPlay: Rewarding the Proposer for Where It Moves the Solvercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data
  2. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
  3. 2026GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games
  4. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
  5. 2025Towards Understanding Self-play for LLM Reasoning
  6. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  7. 2024Can Large Language Models Play Games? A Case Study of A Self-Play Approach
  8. 2026Self-Play Pretraining with Zero Data