mlsanityData to 14 Oct 2026

Self-Play Search Distillation for Large Language Model Reasoning

arXiv 2609.30936 ↗First posted 25 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code also fits: Mathematical reasoning

also in 1 more
Abstract (arXiv)

Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 26 of its 62 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

26 of 62 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning7 May
  2. 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation20 May
  3. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations19 Feb

Other references · 23

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  5. 2026Self-Distillation Enables Continual Learningcited by 247
  6. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  7. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  8. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  9. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  10. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 62
  11. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  12. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 55
11 more
  1. 2024Amortized Planning with Large-Scale Transformers: A Case Study on Chesscited by 49
  2. 2025TextArenacited by 27
  3. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learningcited by 16
  4. 2024Can Large Language Models Play Games? A Case Study of A Self-Play Approachcited by 13
  5. 2025ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Contextcited by 13
  6. 2019Policy Gradient Search: Online Planning and Expert Iteration without Search Treescited by 12
  7. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  8. 2025ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solvingcited by 8
  9. 2025OpenSIR: Open-Ended Self-Improving Reasonercited by 7
  10. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
  11. 2025LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games
  2. 2025TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoning
  3. 2025Towards Understanding Self-play for LLM Reasoning
  4. 2026Scaling Self-Play with Self-Guidance
  5. 2025ChessArena: A Chess Testbed for Evaluating Strategic Reasoning Capabilities of Large Language Models
  6. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning
  7. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
  8. 2024Mastering Board Games by External and Internal Planning with Language Models