mlsanityData to 7 Oct 2026

Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

arXiv 2506.08745 ↗First posted 10 Jun 2025cs.AI cs.CLcited by 43

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (high consistency) with minimal deviation toward other candidates (low volatility). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 73 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

62 of 73 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data25 May
  2. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  5. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  6. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  7. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  8. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization17 Mar
  11. 2025Process-based Self-Rewarding Language Models5 Mar
  12. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
10 more
  1. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  2. 2025Reasoning with Reinforced Functional Token Tuning19 Feb
  3. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models13 Feb
  4. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  6. 2024CREAM: Consistency Regularized Self-Rewarding Language Models16 Oct
  7. 2023Let's Verify Step by Step31 May
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2024Qwen2.5 Technical Reportcited by 4,713
28 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  10. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  11. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  12. 2024Iterative Reasoning Preference Optimizationcited by 244
  13. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  14. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  15. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
  16. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  17. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  18. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  19. 2025Self-rewarding correction for mathematical reasoningcited by 67
  20. 2024Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Comparecited by 66
  21. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
  22. 2025A Survey of Direct Preference Optimizationcited by 35
  23. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  24. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  25. 20242AFC Prompting of Large Multimodal Models for Image Quality Assessmentcited by 20
  26. 2024Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoningcited by 18
  27. 2022Action Candidate Driven Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 8
  28. 2021Action Candidate Based Clipped Double Q-learning for Discrete and Continuous Action Taskscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 30.

43 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  4. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  5. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  6. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  7. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  8. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  9. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 20
  10. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 16
  11. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Timecited by 10
  12. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Modelscited by 7
31 more
  1. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
  2. 2026Tool Verification for Test-Time Reinforcement Learningcited by 6
  3. 2026Retrospective Progress-Aware Self-Refinement for LLM Agent Trainingcited by 6
  4. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 3
  5. 2025BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewardscited by 2
  6. 2026Mental-R1: Aligning LLM Reasoning for Mental Health Assessmentcited by 2
  7. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  8. 2026Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworkscited by 1
  9. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifierscited by 1
  10. 2026RANKVIDEO: Reasoning Reranking for Text-to-Video Retrievalcited by 1
  11. 2026SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Modelscited by 1
  12. 2026InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoningcited by 1
  13. 2026ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Trainingcited by 1
  14. 2026Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoningcited by 1
  15. 2026Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Tracescited by 1
  16. 2026ThoughtFold: Folding Reasoning Chains via Introspective Preference Learningcited by 1
  17. 2026Smart Picks in the Dark: Towards Efficient RLVR for Reasoning via Tracing Metacognitive Pivotscited by 1
  18. 2026GeoMin: Data-Efficient Semi-Supervised RLVR via Geometric Distribution Modelingcited by 1
  19. 2026Learning What to Practice: Diagnosis-Guided Self-Evolution for Language Modelscited by 1
  20. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewardscited by 0
  21. 2026WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvementcited by 0
  22. 2026R-C2: Cycle-Consistent Reinforcement Learning Improves Multimodal Reasoningcited by 0
  23. 2026Know Thy Reasoner: Not All Language Models Explore Alikecited by 0
  24. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  25. 2026Easy Samples Are All You Need: Self-Evolving LLMs via Data-Efficient Reinforcement Learningcited by 0
  26. 2026TimeGuard: Channel-wise Pool Training for Backdoor Defense in Time Series Forecastingcited by 0
  27. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
  28. 2026When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Studycited by 0
  29. 2026Unsupervised Post-Training of Foundation Models: A Surveycited by 0
  30. 2026Adversarial Closed-Loop Curriculum for Evolving Role-Playing Agentscited by 0
  31. 2026Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolvingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  2. 2025Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
  3. 2025Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling
  4. 2026Likelihood-Based Reward Designs for General LLM Reasoning
  5. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  7. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey
  8. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration