mlsanityData to 14 Oct 2026

Learning to Reason without External Rewards

arXiv 2505.19590 ↗First posted 26 May 2025cs.LG cs.CLcited by 210

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Training large language models (LLMs) for complex reasoning via Reinforcement Learning with Verifiable Rewards (RLVR) is effective but limited by reliance on costly, domain-specific supervision. We explore Reinforcement Learning from Internal Feedback (RLIF), a framework that enables LLMs to learn from intrinsic signals without external rewards or labeled data. We propose Intuitor, an RLIF method that uses a model's own confidence-termed self-certainty-as its sole reward signal. Intuitor replaces external rewards in Group Relative Policy Optimization (GRPO) with self-certainty scores, enabling fully unsupervised learning. Experiments demonstrate that Intuitor matches GRPO's performance on mathematical benchmarks while achieving better generalization to out-of-domain tasks like code generation, without requiring gold solutions or test cases. Our findings show that intrinsic model signals can drive effective learning across domains, offering a scalable alternative to RLVR for autonomous AI systems where verifiable rewards are unavailable. Code is available at https://github.com/sunblaze-ucb/Intuitor

Key facts. Cited by 210 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 59 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

41 of 59 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  3. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining12 May
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  5. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  6. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  7. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
16 more
  1. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  4. 2024Self-Rewarding Language Modelscited by 681
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  7. 20242 OLMo 2 Furiouscited by 340
  8. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  9. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  10. 2025Reasoning Models Can Be Effective Without Thinkingcited by 187
  11. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  12. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 87
  13. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  14. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  15. 2024Learning Formal Mathematics From Intrinsic Motivationcited by 45
  16. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 84; 2026: 126.

210 papers.

Sort:

By area:

  1. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  4. 2025Deep Think with Confidencecited by 205
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  6. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  7. 2025Geometric-Mean Policy Optimizationcited by 111
  8. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  9. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  10. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 89
  11. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 85
  12. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contaminationcited by 82
38 more
  1. 2025Can Large Reasoning Models Self-Train?cited by 68
  2. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  3. 2025Self-Questioning Language Modelscited by 50
  4. 2025SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam?cited by 47
  5. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
  6. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  7. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  8. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoningcited by 37
  9. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  10. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  11. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  12. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 35
  13. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  14. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  15. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  16. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  17. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 31
  18. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 29
  19. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  20. 2025RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agentscited by 25
  21. 2025RLP: Reinforcement as a Pretraining Objectivecited by 25
  22. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  23. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimizationcited by 25
  24. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 24
  25. 2025Co-rewarding: Stable Self-supervised RL for Eliciting Reasoning in Large Language Modelscited by 23
  26. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  27. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 19
  28. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  29. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 18
  30. 2026Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gaincited by 17
  31. 2026On-Policy Self-Distillation without Any Supervisioncited by 17
  32. 2025Play to Generalize: Learning to Reason Through Game Playcited by 16
  33. 2025RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalizationcited by 15
  34. 2026Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Informationcited by 14
  35. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  36. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learningcited by 13
  37. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  38. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 12

50 most cited of 210 All 210 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RLSR: Reinforcement Learning from Self Reward
  2. 2026When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study
  3. 2025A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
  4. 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
  5. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards
  6. 2026Momentum for Reasoning: Dense Intrinsic Signals in Policy Optimization
  7. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  8. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs