mlsanityData to 14 Oct 2026

GameSmith: Scaling Zero-Sum Self-Play with Synthesised Games

arXiv 2610.13995 ↗First posted 26 Sep 2026cs.LG cs.AI cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → Self-play RL for reasoning

Abstract (arXiv)

Post-training large language models (LLMs) on two-player zero-sum games provides a verifiable reward, and the gains have been shown to transfer to general reasoning. Existing pipelines, however, rely on a handful of hand-written games and on advantage estimators tied to them, so both the supply of games and the advantage estimate on a large corpus limit scaling. We present GameSmith, which scales both. An LLM synthesises games inside templates whose reward, observation, and hidden-information code is fixed. Each game is verified by seed-paired replay and by an independent re-implementation of its specification, then expanded into encoding, rule, and size variants, giving 26 new games in 29 families and 336 verified variants from the three SPIRAL templates. The model is distilled from the self-play of a larger teacher (Qwen3-32B) on these games and then trained by self-play reinforcement learning with CRAB, an advantage estimator that credits each decision against a shrinkage estimate of the expected return of its position and is defined on every variant. On Qwen3-4B-Base, distillation raises the 13-benchmark mean pass@1 from 45.5 to 62.5, above the three SPIRAL games at the same budget, and self-play with CRAB reaches 64.9. The ordering holds across models, and transfer grows with the number of game families.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Self-play RL for reasoning, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 52 of its 64 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

52 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Learning to Build the Environment: Self-Evolving Reasoning RL via Verifiable Environment Synthesis14 May
  2. 2025MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs17 Oct
  3. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning30 Jun
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  5. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoning27 Apr
  6. 2024Self-playing Adversarial Language Game Enhances LLM Reasoning16 Apr

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
34 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  5. 2024Self-Rewarding Language Modelscited by 681
  6. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  7. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  8. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  9. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  10. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  12. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  13. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 120
  14. 2024How Well Can LLMs Negotiate? NegotiationArena Platform and Analysiscited by 100
  15. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 95
  16. 2021Provably Strict Generalisation Benefit for Equivariant Modelscited by 91
  17. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  18. 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 74
  19. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  20. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 66
  21. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
  22. 2025Self-Challenging Language Model Agentscited by 59
  23. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  24. 2025Self-Questioning Language Modelscited by 50
  25. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  26. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  27. 2025Measuring General Intelligence with Generated Gamescited by 24
  28. 2026GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injectioncited by 11
  29. 2023BERT is not The Count: Learning to Match Mathematical Statements with Proofscited by 10
  30. 2022Benefits of Permutation-Equivariance in Auction Mechanismscited by 8
  31. 2026Spectral Attention Steering for Prompt Highlightingcited by 8
  32. 2026Time is Not a Label: Continuous Phase Rotation for Temporal Knowledge Graphs and Agentic Memorycited by 4
  33. 2026The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillationcited by 3
  34. 2026PRISM: Parallel Reward Integration with Symmetry for MORLcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Play Search Distillation for Large Language Model Reasoning
  2. 2025Language Self-Play For Data-Free Training
  3. 2024Can Large Language Models Play Games? A Case Study of A Self-Play Approach
  4. 2025Towards Understanding Self-play for LLM Reasoning
  5. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
  6. 2026Scaling Self-Play with Self-Guidance
  7. 2024Large Language Models as Agents in Two-Player Games
  8. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Games