mlsanityData to 14 Oct 2026

Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision

arXiv 2411.16579 ↗First posted 25 Nov 2024cs.CL cs.AI cs.LGcited by 60

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Training large language models (LLMs) to spend more time thinking and reflection before responding is crucial for effectively solving complex reasoning tasks in fields such as science, coding, and mathematics. However, the effectiveness of mechanisms like self-reflection and self-correction depends on the model's capacity to accurately assess its own performance, which can be limited by factors such as initial accuracy, question difficulty, and the lack of external feedback. In this paper, we delve into a two-player paradigm that separates the roles of reasoning and critique models, where the critique model provides step-level feedback to supervise the reasoning (actor) model during both test-time and train-time. We first propose AutoMathCritique, an automated and scalable framework for collecting critique data, resulting in a dataset of 76,321 responses paired with step-level feedback. Fine-tuning language models with this dataset enables them to generate natural language feedback for mathematical reasoning. We demonstrate that the critique models consistently improve the actor's performance on difficult queries at test-time, especially when scaling up inference-time computation. Motivated by these findings, we introduce the critique-based supervision to the actor's self-training process, and propose a critique-in-the-loop self-improvement method. Experiments show that the method improves the actor's exploration efficiency and solution diversity, especially on challenging queries, leading to a stronger reasoning model. Lastly, we take the preliminary step to explore training self-talk reasoning models via critique supervision and showcase its potential. Our code and datasets are at hrefhttps://mathcritique.github.io/https://mathcritique.github.io/.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 3 of its 78 references that are in the map. First posted on arXiv 25 Nov 2024.

Builds on

3 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing18 Apr

Other references · 2

  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2024Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Samplingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 48; 2026: 11.

60 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  4. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  5. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  6. 2025ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learningcited by 79
  7. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  8. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  9. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 42
  10. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  11. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Datacited by 40
  12. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
38 more
  1. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
  2. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 37
  3. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 31
  4. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  5. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
  6. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 21
  7. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  8. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 19
  9. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitationcited by 18
  10. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  11. 2025R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systemscited by 16
  12. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  13. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  14. 2026AgentV-RL: Scaling Reward Modeling with Agentic Verifiercited by 15
  15. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  16. 2025TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scalingcited by 14
  17. 2025MetaScale: Test-Time Scaling with Evolving Meta-Thoughtscited by 13
  18. 2025Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critiquecited by 13
  19. 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoningcited by 13
  20. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  21. 2025Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Frameworkcited by 10
  22. 2025J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimizationcited by 9
  23. 2025Mitigating Manipulation and Enhancing Persuasion: A Reflective Multi-Agent Approach for Legal Argument Generationcited by 8
  24. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  25. 2025MedReflect: Teaching Medical LLMs to Self-Improve via Reflective Correctioncited by 8
  26. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Gamecited by 7
  27. 2025Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learningcited by 7
  28. 2025Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systemscited by 6
  29. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solvingcited by 6
  30. 2025Step-Wise Formal Verification for LLM-Based Mathematical Problem Solvingcited by 5
  31. 2025Learning from Natural Language Feedback for Personalized Question Answeringcited by 5
  32. 2026DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMscited by 5
  33. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  34. 2026ThinkTwice: Jointly Optimizing Large Language Models for Reasoning and Self-Refinementcited by 5
  35. 2025Scalable Oversight for Superhuman AI via Recursive Self-Critiquingcited by 4
  36. 2025Making VLMs More Robot-Friendly: Self-Critical Distillation of Low-Level Procedural Reasoningcited by 4
  37. 2025KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integrationcited by 4
  38. 2025Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learningcited by 4

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoning
  2. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate
  3. 2026Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
  4. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoning
  5. 2025Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLM
  6. 2023MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
  7. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic
  8. 2025Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning