# Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning

arXiv: https://arxiv.org/abs/2506.08745 · HTML: https://mlsanity.com/paper/2506.08745/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 62 of its 73 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- Synthetic data and self-improvement for LLMs / [Self-play RL for reasoning](https://mlsanity.com/synth/self-play-rl-for-reasoning/)

## Builds on (most cited first, 30 of 62)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)

## Built on by (most cited first, 30 of 46)

- [R-Zero: Self-Evolving Reasoning LLM from Zero Data](https://mlsanity.com/paper/2508.05004/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning](https://mlsanity.com/paper/2511.16043/) (2025)
- [Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning](https://mlsanity.com/paper/2508.16949/) (2025)
- [Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agents](https://mlsanity.com/paper/2509.09265/) (2025)
- [Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation](https://mlsanity.com/paper/2509.15194/) (2025)
- [Guided Self-Evolving LLMs with Minimal Human Supervision](https://mlsanity.com/paper/2512.02472/) (2025)
- [Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Models](https://mlsanity.com/paper/2508.00410/) (2025)
- [Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain](https://mlsanity.com/paper/2603.02218/) (2026)
- [What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time](https://mlsanity.com/paper/2603.19880/) (2026)
- [Tool Verification for Test-Time Reinforcement Learning](https://mlsanity.com/paper/2603.02203/) (2026)
- [A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models](https://mlsanity.com/paper/2506.18485/) (2025)
- [Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks](https://mlsanity.com/paper/2509.23067/) (2025)
- [When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.25864/) (2026)
- [Retrospective Progress-Aware Self-Refinement for LLM Agent Training](https://mlsanity.com/paper/2606.14302/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality](https://mlsanity.com/paper/2509.23765/) (2025)
- [BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewards](https://mlsanity.com/paper/2510.09596/) (2025)
- [Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning](https://mlsanity.com/paper/2604.21327/) (2026)
- [DARL: Encouraging Diverse Answers for General Reasoning without Verifiers](https://mlsanity.com/paper/2601.14700/) (2026)
- [Mental-R1: Aligning LLM Reasoning for Mental Health Assessment](https://mlsanity.com/paper/2606.13176/) (2026)
- [Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards](https://mlsanity.com/paper/2601.04411/) (2026)
- [Dual-Phase LLM Reasoning: Self-Evolved Mathematical Frameworks](https://mlsanity.com/paper/2601.05616/) (2026)
- [RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval](https://mlsanity.com/paper/2602.02444/) (2026)
- [SPA-Cache: Singular Proxies for Adaptive Caching in Diffusion Language Models](https://mlsanity.com/paper/2602.02544/) (2026)
- [InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning](https://mlsanity.com/paper/2603.17310/) (2026)
- [ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training](https://mlsanity.com/paper/2604.07484/) (2026)
- [Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces](https://mlsanity.com/paper/2605.14358/) (2026)
- [ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning](https://mlsanity.com/paper/2606.03503/) (2026)

## Abstract (arXiv)

Recent advances of Reinforcement Learning (RL) have highlighted its potential in complex reasoning tasks, yet effective training often relies on external supervision, which limits the broader applicability. In this work, we propose a novel self-rewarding reinforcement learning framework to enhance Large Language Model (LLM) reasoning by leveraging the consistency of intermediate reasoning states across different reasoning trajectories. Our key insight is that correct responses often exhibit consistent trajectory patterns in terms of model likelihood: their intermediate reasoning states tend to converge toward their own final answers (high consistency) with minimal deviation toward other candidates (low volatility). Inspired by this observation, we introduce CoVo, an intrinsic reward mechanism that integrates Consistency and Volatility via a robust vector-space aggregation strategy, complemented by a curiosity bonus to promote diverse exploration. CoVo enables LLMs to perform RL in a self-rewarding manner, offering a scalable pathway for learning to reason without external supervision. Extensive experiments on diverse reasoning benchmarks show that CoVo achieves performance comparable to or even surpassing supervised RL. Our code is available at https://github.com/sastpg/CoVo.