mlsanityData to 14 Oct 2026

SPICE: Self-Play In Corpus Environments Improves Reasoning

arXiv 2510.24684 ↗First posted 28 Oct 2025cs.CLcited by 83

Not on the map. Why?

Cited by 33 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general reasoning (+9.8%) benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

52 of 64 references.

References · 52

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2023Let's Verify Step by Stepcited by 3,789
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  11. 2024Self-Rewarding Language Modelscited by 681
  12. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  13. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  14. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  15. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  16. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  17. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  18. 2024MAmmoTH2: Scaling Instructions from the Webcited by 145
  19. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  20. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  21. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  22. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 100
  23. 2019AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligencecited by 86
  24. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  25. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  26. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  27. 2025How much do language models memorize?cited by 61
  28. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  29. 2025Self-Questioning Language Modelscited by 50
  30. 2025Language Self-Play For Data-Free Trainingcited by 43
  31. 2025SPC: Evolving Self-Play Critic via Adversarial Games for LLM Reasoningcited by 42
  32. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
  33. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 38
  34. 2019Emergent Linguistic Phenomena in Multi-Agent Communication Gamescited by 36
  35. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  36. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 29
  37. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  38. 2025Training Language Models for Social Deduction with Multi-Agent Reinforcement Learningcited by 16

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 76.

83 papers.

Sort:

By area:

  1. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  2. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  3. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
  4. 2025Scaling Agent Learning via Experience Synthesiscited by 44
  5. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  6. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 40
  7. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  8. 2026GATES: Self-Distillation under Privileged Context with Consensus Gatingcited by 35
  9. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  10. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  11. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 21
  12. 2026TTCS: Test-Time Curriculum Synthesis for Self-Evolvingcited by 19
38 more
  1. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
  2. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  3. 2026G-Zero: Self-Play for Open-Ended Generation from Zero Datacited by 17
  4. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 16
  5. 2026Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesiscited by 12
  6. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  7. 2026Safe and Scalable Web Agent Learning via Recreated Websitescited by 11
  8. 2026V-Zero: Self-Improving Multimodal Reasoning with Zero Annotationcited by 10
  9. 2026Scaling Self-Play with Self-Guidancecited by 10
  10. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
  11. 2026When to Memorize and When to Stop: Gated Recurrent Memory for Long-Context Reasoningcited by 9
  12. 2026Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skillscited by 9
  13. 2025AI & Human Co-Improvement for Safer Co-Superintelligencecited by 8
  14. 2026Active Zero: Self-Evolving Vision-Language Models through Active Environment Explorationcited by 8
  15. 2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7
  16. 2026π-Play: Multi-Agent Self-Play via Privileged Self-Distillation without External Datacited by 7
  17. 2026SCOPE: Self-Play via Co-Evolving Policies for Open-Ended Taskscited by 7
  18. 2026DARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolutioncited by 6
  19. 2026One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RLcited by 6
  20. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  21. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  22. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 5
  23. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  24. 2026Adaptive Time Series Reasoning via Segment Selectioncited by 4
  25. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  26. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 4
  27. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  28. 2026Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSamplingcited by 3
  29. 2026Training LLM Agents for Spontaneous, Reward-Free Self-Evolution via World Knowledge Explorationcited by 3
  30. 2026PREPING: Zero-Task Agent Memory Construction via Self-Practicecited by 3
  31. 2026EvoVid: Temporal-Centric Self-Evolution for Video Large Language Modelscited by 3
  32. 2026Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoningcited by 3
  33. 2026Socratic-Geo: Synthetic Data Generation and Geometric Reasoning via Multi-Agent Interactioncited by 2
  34. 2026Preventing Curriculum Collapse in Self-Evolving Reasoning Systemscited by 2
  35. 2026Cognitive Training for Language Models: Towards General Capabilities via Cross-Entropy Gamescited by 2
  36. 2026Query-Conditioned Test-Time Self-Training for Large Language Modelscited by 2
  37. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Playcited by 2
  38. 2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluationcited by 1

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025OpenSIR: Open-Ended Self-Improving Reasoner
  2. 2025Towards Understanding Self-play for LLM Reasoning
  3. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  4. 2026Self-Play Search Distillation for Large Language Model Reasoning
  5. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL
  6. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
  7. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
  8. 2025Better LLM Reasoning via Dual-Play