mlsanityData to 14 Oct 2026

Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

arXiv 2506.03106 ↗First posted 3 Jun 2025cs.CL cs.AIcited by 90

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs). However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures. We show that plateaued RL models can successfully refine failed solutions when given natural language critiques. Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization. This approach enables LLMs to learn simultaneously from initial responses and critique-guided refinements, effectively internalizing the exploration benefits of both stages. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024. The code and models are released at: https://github.com/zhangxy-2019/critique-GRPO

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 47 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

33 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  2. 2025Learning to Reason under Off-Policy Guidance21 Apr
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2023Let's Verify Step by Step31 May
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024GPT-4o System Cardcited by 4,588
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  11. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  12. 2023TheoremQA: A Theorem-driven Question Answering datasetcited by 192
13 more
  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  2. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  3. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 80
  4. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  5. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 63
  6. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervisioncited by 60
  7. 2023SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Promptingcited by 39
  8. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  9. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  10. 2025ThinkLess: A Training-Free Inference-Efficient Method for Reducing Reasoning Redundancycited by 17
  11. 2024Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answeringcited by 13
  12. 2022Toward Self-learning End-to-End Task-Oriented Dialog Systemscited by 9
  13. 2025Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognitioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 61.

90 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 56
  3. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  4. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  5. 2026Gen-Searcher: Reinforcing Agentic Search for Image Generationcited by 30
  6. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 26
  7. 2025Nudging the Boundaries of LLM Reasoningcited by 26
  8. 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editorcited by 25
  9. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  10. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
  11. 2026Unleashing VLA Potentials in Autonomous Driving via Explicit Learning from Failurescited by 19
  12. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 18
38 more
  1. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  2. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 17
  3. 2026Exploring Reasoning Reward Model for Agentscited by 15
  4. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoningcited by 14
  5. 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoningcited by 13
  6. 2026Rethinking Reward Supervision: Rubric-Conditioned Self-Distillationcited by 13
  7. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  8. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  9. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 10
  10. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  11. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  12. 2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 7
  13. 2025From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMscited by 6
  14. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  15. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 6
  16. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  17. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasonerscited by 5
  18. 2026Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learningcited by 5
  19. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinementcited by 5
  20. 2026PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimizationcited by 5
  21. 2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevancecited by 4
  22. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  23. 2025CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?cited by 4
  24. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 4
  25. 2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loopcited by 4
  26. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  27. 2026ICRL: Learning to Internalize Self-Critique with Reinforcement Learningcited by 4
  28. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  29. 2026Credit Assignment with Resets in Language Model Reasoningcited by 4
  30. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 4
  31. 2025Med-REFL: Medical Reasoning Enhancement via Self-Corrected Fine-grained Reflectioncited by 3
  32. 2025Reducing Cognitive Overhead in Tool Use via Multi-Small-Agent Reinforcement Learningcited by 3
  33. 2026M^4olGen: Multi-Agent, Multi-Stage Molecular Generation under Precise Multi-Property Constraintscited by 3
  34. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidancecited by 3
  35. 2026StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Visioncited by 3
  36. 2026P^2O: Joint Policy and Prompt Optimizationcited by 3
  37. 2026R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoningcited by 3
  38. 2026Find, Fix, Reason: Context Repair for Video Reasoningcited by 3

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  2. 2025LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
  3. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  4. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  5. 2025Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning
  6. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  7. 2025Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning
  8. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning