mlsanityData to 7 Oct 2026

Self-rewarding correction for mathematical reasoning

arXiv 2502.19613 ↗First posted 26 Feb 2025cs.AI cs.LGcited by 67

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

We study self-rewarding reasoning large language models (LLMs), which can simultaneously generate step-by-step reasoning and evaluate the correctness of their outputs during the inference time-without external feedback. This integrated approach allows a single model to independently guide its reasoning process, offering computational advantages for model deployment. We particularly focus on the representative task of self-correction, where models autonomously detect errors in their responses, revise outputs, and decide when to terminate iterative refinement loops. To enable this, we propose a two-staged algorithmic framework for constructing self-rewarding reasoning models using only self-generated data. In the first stage, we employ sequential rejection sampling to synthesize long chain-of-thought trajectories that incorporate both self-rewarding and self-correction mechanisms. Fine-tuning models on these curated data allows them to learn the patterns of self-rewarding and self-correction. In the second stage, we further enhance the models' ability to assess response accuracy and refine outputs through reinforcement learning with rule-based signals. Experiments with Llama-3 and Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction capabilities and achieves performance comparable to systems that rely on external reward models.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 70 of its 88 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

70 of 88 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  2. 2023LLMs cannot find reasoning errors, but can correct them given the error location14 Nov
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  4. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies6 Aug
  5. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 65

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  13. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  14. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  15. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  16. 2024Self-Rewarding Language Modelscited by 654
  17. 2025Process Reinforcement through Implicit Rewardscited by 478
  18. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  19. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  20. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  21. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  22. 2023Language Models can Solve Computer Taskscited by 371
  23. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  24. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  25. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  26. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  27. 2022Generating Sequences by Learning to Self-Correctcited by 271
  28. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  29. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  30. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  31. 2024Iterative Reasoning Preference Optimizationcited by 244
  32. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  33. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  34. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  35. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  36. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  37. 2024Generative Reward Modelscited by 148
  38. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 41; 2026: 26.

67 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  3. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivizationcited by 124
  4. 2025Process Reward Models That Thinkcited by 118
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  6. 2025Scaling Test-time Compute for LLM Agentscited by 59
  7. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  8. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  9. 2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 50
  10. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  11. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
  12. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
38 more
  1. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  2. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 32
  3. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  4. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  5. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  6. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  7. 2025Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RLcited by 23
  8. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  9. 2025Boosting LLM Reasoning via Spontaneous Self-Correctioncited by 18
  10. 2025Meta-RL Induces Exploration in Language Agentscited by 17
  11. 2025ReVeal: Self-Evolving Code Agents via Reliable Self-Verificationcited by 16
  12. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  13. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  14. 2025World Modelling Improves Language Model Agentscited by 12
  15. 2025Surrogate Signals from Format and Length: Reinforcement Learning for Solving Mathematical Problems without Ground Truth Answerscited by 11
  16. 2025Sherlock: Self-Correcting Reasoning in Vision-Language Modelscited by 11
  17. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  18. 2026On-Policy Self-Distillation without Any Supervisioncited by 11
  19. 2025What Defines Good Reasoning in LLMs? Dissecting Reasoning Steps with Multi-Aspect Evaluationcited by 10
  20. 2025Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld's Episode Theorycited by 9
  21. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolutioncited by 9
  22. 2025Schoenfeld's Anatomy of Mathematical Reasoning by Language Modelscited by 8
  23. 2026Meta-Reinforcement Learning with Self-Reflection for Agentic Searchcited by 8
  24. 2026Understanding the Challenges in Iterative Generative Optimization with LLMscited by 8
  25. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  26. 2025From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMscited by 6
  27. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoningcited by 6
  28. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solvingcited by 6
  29. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  30. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinementcited by 5
  31. 2025Thinking Machines: Mathematical Reasoning in the Age of LLMscited by 4
  32. 2025From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Trainingcited by 4
  33. 2025LightReasoner: Can Small Language Models Teach Large Language Models Reasoning?cited by 3
  34. 2025Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learningcited by 3
  35. 2026ACE: Self-Evolving LLM Coding Framework via Adversarial Unit Test Generation and Preference Optimizationcited by 3
  36. 2026Can LLMs Learn to Reason Robustly under Noisy Supervision?cited by 2
  37. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  38. 2026Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Predictioncited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Incentivizing LLMs to Self-Verify Their Answers
  2. 2025Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
  3. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning
  4. 2025Language Models can perform Single-Utterance Self-Correction of Perturbed Reasoning
  5. 2024Embedding Self-Correction as an Inherent Ability in Large Language Models for Enhanced Mathematical Reasoning
  6. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  7. 2025Process-based Self-Rewarding Language Models
  8. 2024Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning