mlsanityData to 14 Oct 2026

Incentivizing LLMs to Self-Verify Their Answers

arXiv 2506.01369 ↗First posted 2 Jun 2025cs.LG cs.AIcited by 17

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable progress in complex reasoning tasks through both post-training and test-time scaling laws. While prevalent test-time scaling approaches are often realized by using external reward models to guide the model generation process, we find that only marginal gains can be acquired when scaling a model post-trained on specific reasoning tasks. We identify that the limited improvement stems from distribution discrepancies between the specific post-trained generator and the general reward model. To address this, we propose a framework that incentivizes LLMs to self-verify their own answers. By unifying answer generation and verification within a single reinforcement learning (RL) process, we train models that can effectively assess the correctness of their own solutions. The trained model can further scale its performance at inference time by verifying its generations, without the need for external verifiers. We train our self-verification models based on Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, demonstrating their capabilities across varying reasoning context lengths. Experiments on multiple mathematical reasoning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 45 of its 66 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

45 of 66 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking25 Mar
  2. 2025s1: Simple test-time scaling31 Jan
  3. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  5. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning26 Apr
  6. 2024V-STaR: Training Verifiers for Self-Taught Reasoners9 Feb
  7. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training29 Sep
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  9. 2023Self-Evaluation Guided Beam Search for Reasoning1 May
  10. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  11. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2023Let's Verify Step by Stepcited by 3,789
22 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  7. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  8. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  13. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  14. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  15. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  16. 2024More Agents Is All You Needcited by 180
  17. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  18. 2024Generative Reward Modelscited by 148
  19. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  20. 2024The Limits of Inference Scaling Through Resamplingcited by 53
  21. 2025One-shot Entropy Minimizationcited by 32
  22. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifierscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  2. 2026Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMscited by 10
  3. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasonerscited by 5
  4. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinementcited by 5
  5. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learningcited by 4
  6. 2026iGRPO: Self-Feedback-Driven LLM Reasoningcited by 2
  7. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
  8. 2026MAVEN: Multi-Agent Verification-Elaboration Network with In-Step Epistemic Auditingcited by 1
  9. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verificationcited by 1
  10. 2026Two-Stage Reinforcement Learning for Sound and Adversarial Test Generation in Code LLMscited by 1
  11. 2026AIPO: Learning to Reason from Active Interactioncited by 0
  12. 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scalingcited by 0
5 more
  1. 2026Guarded Repair for Harm-Aware Post-hoc Replacement of LLM Mathematical Reasoningcited by 0
  2. 2026FABSVer: Faster Training and Better Self-Verification for LLM Mathematical Reasoningcited by 0
  3. 2026REAR: Test-time Preference Realignment through Reward Decompositioncited by 0
  4. 2026Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7Bcited by 0
  5. 2026Data-Centric Recursive Self-Improvement for Navigation via Policy-Synchronized Language-Action Dual Bootstrappingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Self-rewarding correction for mathematical reasoning
  2. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  3. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
  4. 2025From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
  5. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  6. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  7. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling
  8. 2025TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning