mlsanityData to 7 Oct 2026

Towards Understanding Self-play for LLM Reasoning

arXiv 2510.27072 ↗First posted 31 Oct 2025cs.LGcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advances in large language model (LLM) reasoning, led by reinforcement learning with verifiable rewards (RLVR), have inspired self-play post-training, where models improve by generating and solving their own problems. While self-play has shown strong in-domain and out-of-domain gains, the mechanisms behind these improvements remain poorly understood. In this work, we analyze the training dynamics of self-play through the lens of the Absolute Zero Reasoner, comparing it against RLVR and supervised fine-tuning (SFT). Our study examines parameter update sparsity, entropy dynamics of token distributions, and alternative proposer reward functions. We further connect these dynamics to reasoning performance using pass@k evaluations. Together, our findings clarify how self-play differs from other post-training strategies, highlight its inherent limitations, and point toward future directions for improving LLM math reasoning through self-play.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 28 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

26 of 28 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR19 Aug
  2. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy16 Jun
  5. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  6. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  9. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  12. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
3 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  4. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 182
  5. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  6. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 115
  7. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  8. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  9. 2025Self-Questioning Language Modelscited by 47
  10. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  11. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  2. 2026Scaling Self-Play with Self-Guidancecited by 9
  3. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 7
  4. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  5. 2026Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolutioncited by 1
  6. 2026INFUSER: Influence-Guided Self-Evolution Improves Reasoningcited by 0
  7. 2026Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generationcited by 0
  8. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  2. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
  3. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
  4. 2025LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning
  5. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
  6. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  7. 2025Incentivizing LLMs to Self-Verify Their Answers
  8. 2025OpenSIR: Open-Ended Self-Improving Reasoner