mlsanityData to 14 Oct 2026

Teaching Language Models to Critique via Reinforcement Learning

arXiv 2502.03492 ↗First posted 5 Feb 2025cs.LG cs.AI cs.CLcited by 42

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose textttCTRL, a framework for textttCritic textttTraining via textttReinforcement textttLearning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with textttCTRL significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 47 of its 65 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

47 of 65 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec
  2. 2024What Makes Large Language Models Reason in (Multi-Turn) Code Generation?10 Oct
  3. 2024Qwen2.5-Coder Technical Report18 Sep
  4. 2024LLM Critics Help Catch LLM Bugs28 Jun
  5. 2024NExT: Teaching Large Language Models to Reason about Code Execution23 Apr
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  7. 2023Teaching Large Language Models to Self-Debug11 Apr
  8. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
27 more
  1. 2024Self-Rewarding Language Modelscited by 681
  2. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  4. 2018AI safety via debatecited by 400
  5. 2022Self-critiquing models for assisting human evaluatorscited by 380
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  7. 2022Generating Sequences by Learning to Self-Correctcited by 277
  8. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  9. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  10. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  11. 2018Supervising strong learners by amplifying weak expertscited by 145
  12. 2022Learning by Distilling Contextcited by 138
  13. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  14. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  15. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 112
  16. 2024Critique-out-Loud Reward Modelscited by 106
  17. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  18. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  19. 2023Shepherd: A Critic for Language Model Generationcited by 97
  20. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  21. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
  22. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 60
  23. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  24. 2024FullStack Bench: Evaluating LLMs as Full Stack Coderscited by 58
  25. 2024Improving Reward Models with Synthetic Critiquescited by 44
  26. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 26
  27. 2024Natural Language Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 26; 2026: 16.

42 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  4. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  6. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 124
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  8. 2025GenPRM: Scaling Test-Time Compute of Process Reward Models via Generative Reasoningcited by 97
  9. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  10. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  11. 2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 66
  12. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
30 more
  1. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  2. 2025ReVeal: Self-Evolving Code Agents via Reliable Self-Verificationcited by 18
  3. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  4. 2025Training Language Models to Generate Quality Code with Program Analysis Feedbackcited by 12
  5. 2026Bringing Value Models Back: Generative Critics for Value Modeling in LLM Reinforcement Learningcited by 11
  6. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  7. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  8. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
  9. 2025Neural Theorem Proving: Generating and Structuring Proofs for Formal Verificationcited by 5
  10. 2025Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learningcited by 4
  11. 2026MaxCode: A Max-Reward Reinforcement Learning Framework for Automated Code Optimizationcited by 4
  12. 2026Teaching Large Reasoning Models Effective Reflectioncited by 4
  13. 2026DICE: Diffusion Large Language Models Excel at Generating CUDA Kernelscited by 3
  14. 2025Critique-Guided Distillation for Robust Reasoning via Refinementcited by 2
  15. 2025Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuningcited by 2
  16. 2025Orchestrator: Active Inference for Multi-Agent Systems in Long-Horizon Taskscited by 2
  17. 2026RM -RF: Reward Model for Run-Free Unit Test Evaluationcited by 2
  18. 2026Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decisioncited by 2
  19. 2026From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memorycited by 2
  20. 2026CAFE: Self-Improving Search Agents Need Co-Evolving Feedbackcited by 2
  21. 2025SELF-REDRAFT: Eliciting Intrinsic Exploration-Exploitation Balance in Test-Time Scaling for Code Generationcited by 1
  22. 2025Rectify Evaluation Preference: Improving LLMs' Critique on Math Reasoning via Perplexity-aware Reinforcement Learningcited by 1
  23. 2026Internalizing Agency from Reflective Experiencecited by 1
  24. 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoningcited by 1
  25. 2026Primal Generation, Dual Judgment: Self-Training from Test-Time Scalingcited by 0
  26. 2026Test-Time Hinting for Black-Box Vision-Language Modelscited by 0
  27. 2026VeriFine: Scaling Verification for Self-Improvement in Embodied Reasoningcited by 0
  28. 2026Verify Less, Evolve More: Training Idea-Level Critics for Verification-Efficient ML Evolving Agentscited by 0
  29. 2026One-Sided Verification Under Limited Expert Reviewcited by 0
  30. 2026RLPE: Teaching LLMs to Prompt with Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Critic-Guided Decoding for Controlled Text Generation
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  3. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
  4. 2025Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning
  5. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation
  6. 2023Improving Code Generation by Training with Natural Language Feedback
  7. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  8. 2026ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning