mlsanityData to 7 Oct 2026

Training Language Models to Self-Correct via Reinforcement Learning

arXiv 2409.12917 ↗First posted 19 Sep 2024cs.LGcited by 405

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Current methods for training self-correction typically depend on either multiple models, a more advanced model, or additional forms of supervision. To address these shortcomings, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are often insufficient for instilling self-correction behavior. In particular, we observe that training via SFT falls prey to either a distribution mismatch between mistakes made by the data-collection policy and the model's own responses, or to behavior collapse, where learning implicitly prefers only a certain mode of correction behavior that is often not effective at self-correction on test problems. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction behavior that is effective at test time as opposed to fitting high-reward responses for a given prompt. This regularization process includes an initial phase of multi-turn RL on a base model to generate a policy initialization that is less susceptible to collapse, followed by using a reward bonus to amplify self-correction. With Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.

Key facts. Cited by 405 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 50 of its 58 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

50 of 58 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning26 Apr
  3. 2023LLMs cannot find reasoning errors, but can correct them given the error location14 Nov
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  5. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies6 Aug
  6. 2023REFINER: Reasoning Feedback on Intermediate Representations4 Apr
  7. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  8. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 42

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
30 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  3. 2022Self-critiquing models for assisting human evaluatorscited by 372
  4. 2023Language Models can Solve Computer Taskscited by 371
  5. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  6. 2022Generating Sequences by Learning to Self-Correctcited by 271
  7. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  10. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  11. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  12. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  13. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  14. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  15. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  16. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  17. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  18. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 125
  19. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  20. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  21. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  22. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 101
  23. 2019Meta-Learning without Memorizationcited by 92
  24. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  25. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  26. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  27. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  28. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 51
  29. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  30. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 237; 2026: 135.

405 papers.

Sort:

By area:

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  2. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  3. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  4. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  5. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  6. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 308
  7. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  8. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  9. 2025Training Language Models to Reason Efficientlycited by 242
  10. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  11. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  12. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 172
38 more
  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  3. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scalingcited by 135
  4. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  5. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  6. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 121
  7. 2025Process Reward Models That Thinkcited by 118
  8. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  9. 2025SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Taskscited by 108
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  11. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  12. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 106
  13. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  14. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 97
  15. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  16. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 96
  17. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 95
  18. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 91
  19. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  20. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  21. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  22. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  23. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  24. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  25. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 78
  26. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  27. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  28. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 72
  29. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  30. 2025ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoningcited by 70
  31. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  32. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
  33. 2025Self-rewarding correction for mathematical reasoningcited by 67
  34. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  35. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  36. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 62
  37. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  38. 2025Scaling Test-time Compute for LLM Agentscited by 59

50 most cited of 405 All 405 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
  2. 2023Language Model Self-improvement by Reinforcement Learning Contemplation
  3. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  4. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  5. 2024A Theoretical Understanding of Self-Correction through In-context Alignment
  6. 2024Large Language Models have Intrinsic Self-Correction Ability
  7. 2025Towards a Unified View of Large Language Model Post-Training
  8. 2025Behavior Injection: Preparing Language Models for Reinforcement Learning