mlsanityData to 7 Oct 2026

Can Large Reasoning Models Self-Train?

arXiv 2505.21444 ↗First posted 27 May 2025cs.LGcited by 66

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent successes of reinforcement learning (RL) in training large reasoning models motivate the question of whether self-training - the process where a model learns from its own judgments - can be sustained within RL. In this work, we study this question using majority voting as a simple self-feedback mechanism. On a comprehensive set of experiments on both synthetic and real reasoning tasks, we find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration, driving further model improvement. Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking where models learn to maximize training (pseudo-)reward, resulting in sudden and complete performance collapse. Together, these results highlight feedback design as the central challenge and call for future research on mechanisms to enable prolonged self-improvement.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 96 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

42 of 96 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Learning to Reason without External Rewards26 May
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  4. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  8. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models24 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
4 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Prediction27 Aug
  2. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements13 Feb
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2016Concrete Problems in AI Safetycited by 1,582
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2024OpenAI o1 System Cardcited by 1,220
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
14 more
  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  2. 2024Iterative Reasoning Preference Optimizationcited by 244
  3. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  4. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  5. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  6. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  7. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  8. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  9. 2024Self-Consistency Preference Optimizationcited by 50
  10. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 34
  11. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  12. 2025Training a Generally Curious Agentcited by 29
  13. 2019Neural networks are a priori biased towards Boolean functions with low entropycited by 24
  14. 2018Random deep neural networks are biased towards simple functionscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 44.

66 papers.

Sort:

By area:

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  4. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 81
  5. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  6. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  7. 2025Self-Questioning Language Modelscited by 47
  8. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  9. 2025Self-Improving LLM Agents at Test-Timecited by 30
  10. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  11. 2025EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewardscited by 24
  12. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
38 more
  1. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  2. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 13
  3. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  4. 2025Certified Self-Consistency: Statistical Guarantees and Test-Time Training for Reliable Reasoning in LLMscited by 12
  5. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  6. 2025Self-Evolving Vision-Language Models for Image Quality Assessment via Voting and Rankingcited by 10
  7. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Timecited by 10
  8. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervisioncited by 9
  9. 2026PACE: Anytime-Valid Acceptance Tests for Self-Evolving Agentscited by 8
  10. 2025Wisdom of the Crowd: Reinforcement Learning from Coevolutionary Collective Feedbackcited by 6
  11. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  12. 2025Correct Reasoning Paths Visit Shared Decision Pivotscited by 5
  13. 2026Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluationcited by 5
  14. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  15. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  16. 2026When Models Judge Themselves: Unsupervised Self-Evolution for Multimodal Reasoningcited by 4
  17. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RLcited by 4
  18. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  19. 2026Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Trainingcited by 4
  20. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  21. 2026PRISM: A Unified Framework for Post-Training LLMs Without Verifiable Rewardscited by 3
  22. 2026ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignmentcited by 3
  23. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  24. 2026Stabilizing Unsupervised Self-Evolution of MLLMs via Continuous Softened Retracing reSamplingcited by 3
  25. 2026CoAct: Co-Active LLM Preference Learning with Human-AI Synergycited by 3
  26. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matchingcited by 2
  27. 2026Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Rewardcited by 2
  28. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  29. 2025Breaking the Self-Confirming Loop: Diagnosing and Mitigating Systemic Reward Bias in Self-Rewarding RLcited by 1
  30. 2026Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworkscited by 1
  31. 2026Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanismcited by 1
  32. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  33. 2026When Can LLMs Learn to Reason with Weak Supervision?cited by 1
  34. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1
  35. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  36. 2026GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modelingcited by 1
  37. 2026Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Modelscited by 1
  38. 2025Graph-of-Causal Evolution: Challenging Chain-of-Model for Reasoningcited by 0

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Teaching Large Reasoning Models Effective Reflection
  2. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures
  3. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
  4. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  5. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  6. 2025Self-rewarding correction for mathematical reasoning
  7. 2025Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
  8. 2025Incentivizing LLMs to Self-Verify Their Answers