# Self-Play Search Distillation for Large Language Model Reasoning

arXiv: https://arxiv.org/abs/2609.30936 · HTML: https://mlsanity.com/paper/2609.30936/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 26 of its 62 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- Synthetic data and self-improvement for LLMs / [Self-play RL for reasoning](https://mlsanity.com/synth/self-play-rl-for-reasoning/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://mlsanity.com/paper/2505.03335/) (2025)
- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations](https://mlsanity.com/paper/2402.12348/) (2024)
- [Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models](https://mlsanity.com/paper/2403.15498/) (2024)
- [Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles](https://mlsanity.com/paper/2505.19914/) (2025)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [Amortized Planning with Large-Scale Transformers: A Case Study on Chess](https://mlsanity.com/paper/2402.04494/) (2024)
- [TextArena](https://mlsanity.com/paper/2504.11442/) (2025)
- [RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning](https://mlsanity.com/paper/2505.14140/) (2025)
- [Can Large Language Models Play Games? A Case Study of A Self-Play Approach](https://mlsanity.com/paper/2403.05632/) (2024)
- [ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context](https://mlsanity.com/paper/2507.00417/) (2025)
- [Policy Gradient Search: Online Planning and Expert Iteration without Search Trees](https://mlsanity.com/paper/1904.03646/) (2019)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [ToTRL: Unlock LLM Tree-of-Thoughts Reasoning Potential through Puzzles Solving](https://mlsanity.com/paper/2505.12717/) (2025)
- [KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation](https://mlsanity.com/paper/2505.14552/) (2025)
- [OpenSIR: Open-Ended Self-Improving Reasoner](https://mlsanity.com/paper/2511.00602/) (2025)
- [SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Tasks](https://mlsanity.com/paper/2605.31433/) (2026)
- [LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2506.04821/) (2025)
- [Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning](https://mlsanity.com/paper/2605.06840/) (2026)

## Abstract (arXiv)

Improving reasoning abilities in Large Language Models (LLMs) requires high-quality data that exposes difficult decisions, competing alternatives, and their consequences. Data scarcity is driven by the low quality of synthetic data and the cost of human labeling. We introduce Self-Play Search Distillation (SPSD), a framework for generating superhuman synthetic data via self-play of MuZero-like networks trained on board games. SPSD uses executable environments to turn search into structured reasoning problems. At each state, the expert identifies a preferred decision, plausible alternatives, plausible opponent replies, and value estimates. By converting the self-play search records into superhuman chains-of-thought, we train LLMs with environment-grounded supervision. Although trained only on self-play search records, SPSD transfers to unseen mathematics. On Qwen3-4B-Base, it raises the mean over six mathematics benchmarks from 24.1 to 36.6 while increasing the held-out-game win rate from 15% to 45%. SPSD offers an annotation-efficient way to create high-quality synthetic data for improving LLM performance in reasoning tasks.