mlsanityData to 14 Oct 2026

On the Generalization Gap in Self-Evolving Language Model Reasoning

arXiv 2606.01075 ↗First posted 31 May 2026cs.CLcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the self-evolution algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework: single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that self-evolution consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non-trivial gap to oracle supervision. We find that multi-turn critic-revision with large models can reach strong self-evolution performance, with Gemma 12B nearly matching oracle-supervised training. Beyond Knights and Knaves, we also evaluate self-evolution on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop self-evolution can help and show how internally generated supervision remains insufficient under this minimal formulation.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 53 of its 72 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

53 of 72 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Self-Training Large Language Models with Confident Reasoning23 May
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?31 Mar
  3. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers27 Feb
  4. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification20 Feb
  5. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimal17 Feb
  6. 2025Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization8 Feb
  7. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep

Other references · 46

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  9. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  10. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  11. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  12. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
34 more
  1. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  3. 2025Learning to Reason without External Rewardscited by 210
  4. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  6. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 128
  7. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  8. 2026Learning to Discover at Test Timecited by 115
  9. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  10. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  11. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  12. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  13. 2025ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agentscited by 84
  14. 2025Can Large Reasoning Models Self-Train?cited by 68
  15. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  16. 2025Language Self-Play For Data-Free Trainingcited by 43
  17. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
  18. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
  19. 2024CREAM: Consistency Regularized Self-Rewarding Language Modelscited by 35
  20. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  21. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
  22. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 24
  23. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  24. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 18
  25. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  26. 2025Unsupervised Elicitation of Language Modelscited by 11
  27. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  28. 2024LLMs Could Autonomously Learn Without External Supervisioncited by 6
  29. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  30. 2025Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedbackcited by 6
  31. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  32. 2025Latent Veracity Inference for Identifying Errors in Stepwise Reasoningcited by 3
  33. 2025Self-CriTeach: LLM Self-Teaching and Self-Critiquing for Improving Robotic Planning via Automated Domain Generationcited by 2
  34. 2026Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesiscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 32
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  3. 2026Phantom Gains: Auditing Self-Improvement Against a Measured Nullcited by 2
  4. 2026Vestrum: Improving Agent Harnesses by Adapting Their Verification, Structure and Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Self-Evolve
  2. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  3. 2026Learning to Self-Verify Makes Language Models Better Reasoners
  4. 2024A Survey on Self-Evolution of Large Language Models
  5. 2026When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain
  6. 2025DEBATE, TRAIN, EVOLVE: Self Evolution of Language Model Reasoning
  7. 2026AERO: Autonomous Evolutionary Reasoning Optimization via Endogenous Dual-Loop Feedback
  8. 2026When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoning