Data to 14 Oct 2026

SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning

arXiv 2506.24119 ↗First posted 30 Jun 2025cs.AI cs.CL cs.LGcited by 100

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more

Abstract (arXiv)

Recent advances in reinforcement learning have shown that language models can develop sophisticated reasoning through training on tasks with verifiable rewards, but these approaches depend on human-curated problem-answer pairs and domain-specific reward engineering. We introduce SPIRAL, a self-play framework where models learn by playing multi-turn, zero-sum games against continuously improving versions of themselves, generating an automatic curriculum of stronger opponents, and eliminating the need for human supervision. To enable this self-play training at scale, we implement a fully online, multi-turn, multi-agent reinforcement learning system for LLMs and propose role-conditioned advantage estimation (RAE) to stabilize multi-agent training. SPIRAL produces reasoning capabilities that transfer broadly, improving performance by up to 10% across a suite of 8 reasoning benchmarks on 4 different models spanning Qwen and Llama model families, outperforming supervised fine-tuning on 25,000 expert game trajectories. Multi-game training (TicTacToe, Kuhn Poker, Simple Negotiation) yields the strongest results, with improvements observed across both base and instruction-tuned models. Analysis of chain-of-thought traces reveals that games develop distinct cognitive patterns that transfer to improve reasoning performance, with different games developing complementary strengths. Even models which have already been trained on reasoning tasks using RLVR, like DeepSeek-R1-Distill-Qwen-7B, still benefit from our approach. These results demonstrate that zero-sum games naturally develop transferable reasoning capabilities across diverse model architectures and training stages, highlighting a promising direction for autonomous reasoning development. Our code can be found in https://github.com/spiral-rl/spiral.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 52 of its 71 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

52 of 71 references.

Closest ancestors · 8

Same direction, newest first

Other references · 44

2022Training language models to follow instructions with human feedbackcited by 15,971
2024The Llama 3 Herd of Modelscited by 14,715
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2025Qwen3 Technical Reportcited by 8,820
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2023Let's Verify Step by Stepcited by 3,789
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392

32 more

2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
2022Solving math word problems with process- and outcome-based feedbackcited by 901
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
2024Self-Rewarding Language Modelscited by 681
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 197
2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
2024Prover-Verifier Games improve legibility of LLM outputscited by 85
2025Reinforcing General Reasoning without Verifierscited by 77
2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
2019Emergent Linguistic Phenomena in Multi-Agent Communication Gamescited by 36
2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
2024Natural Language Reinforcement Learningcited by 17
2025Training Language Models for Social Deduction with Multi-Agent Reinforcement Learningcited by 16
2025Play to Generalize: Learning to Reason Through Game Playcited by 16
2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 15
2024Boundless Socratic Learning with Language Gamescited by 12
2025Divide-Fuse-Conquer: Eliciting "Aha Moments" in Multi-Scenario Gamescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 74.

100 papers.

Sort:

By area:

2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
2025Multi-Agent Evolve: LLM Self-Improve through Co-evolutioncited by 60
2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
2025LLM Collaboration With Multi-Agent Reinforcement Learningcited by 49
2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
2025Language Self-Play For Data-Free Trainingcited by 43
2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37

38 more

2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 31
2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 29
2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMscited by 28
2025On Predictability of Reinforcement Learning Dynamics for Large Language Modelscited by 23
2025GEM: A Gym for Agentic LLMscited by 23
2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 23
2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 19
2026From Context to Skills: Can Language Models Learn from Context Skillfully?cited by 19
2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
2026On-Policy Self-Distillation without Any Supervisioncited by 17
2025MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMscited by 16
2026Position: Agentic Evolution is the Path to Evolving LLMscited by 16
2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotationcited by 10
2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Trainingcited by 10
2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
2026Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skillscited by 9
2025Multi-Actor Generative Artificial Intelligence as a Game Enginecited by 8
2025Towards Understanding Self-play for LLM Reasoningcited by 8
2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 8
2026RelayLLM: Efficient Reasoning via Collaborative Decodingcited by 8
2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 8
2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
2025OpenSIR: Open-Ended Self-Improving Reasonercited by 7
2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7
2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
2026Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewardscited by 7
2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 6
2026Generative AI collective behavior needs an interactionist paradigmcited by 6
2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 6
2026SAGE: Multi-Agent Self-Evolution for LLM Reasoningcited by 6
2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 6
2026Epistemic Gain, Aleatoric Cost: Uncertainty Decomposition in Multi-Agent Debate for Math Reasoningcited by 5
2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Gamescited by 5

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way