mlsanityData to 7 Oct 2026

CriticBench: Benchmarking LLMs for Critique-Correct Reasoning

arXiv 2402.14809 ↗First posted 22 Feb 2024cs.CL cs.AI cs.LGcited by 88

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

The ability of Large Language Models (LLMs) to critique and refine their reasoning is crucial for their application in evaluation, feedback provision, and self-improvement. This paper introduces CriticBench, a comprehensive benchmark designed to assess LLMs' abilities to critique and rectify their reasoning across a variety of tasks. CriticBench encompasses five reasoning domains: mathematical, commonsense, symbolic, coding, and algorithmic. It compiles 15 datasets and incorporates responses from three LLM families. Utilizing CriticBench, we evaluate and dissect the performance of 17 LLMs in generation, critique, and correction reasoning, i.e., GQC reasoning. Our findings reveal: (1) a linear relationship in GQC capabilities, with critique-focused training markedly enhancing performance; (2) a task-dependent variation in correction effectiveness, with logic-oriented tasks being more amenable to correction; (3) GQC knowledge inconsistencies that decrease as model size increases; and (4) an intriguing inter-model critiquing dynamic, where stronger models are better at critiquing weaker ones, while weaker models can surprisingly surpass stronger ones in their self-critique. We hope these insights into the nuanced critique-correct reasoning of LLMs will foster further research in LLM critique and self-improvement.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 47 of its 63 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

47 of 63 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  2. 2023Shepherd: A Critic for Language Model Generation8 Aug
  3. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
  4. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
31 more
  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  13. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  14. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  15. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  16. 2023Teaching Large Language Models to Self-Debugcited by 963
  17. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  18. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  19. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  20. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  21. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  22. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  23. 2022Self-critiquing models for assisting human evaluatorscited by 372
  24. 2023Language Models can Solve Computer Taskscited by 371
  25. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  26. 2022Generating Sequences by Learning to Self-Correctcited by 271
  27. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  28. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  29. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
  30. 2024SciAgent: Tool-augmented Language Models for Scientific Reasoningcited by 69
  31. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 50; 2026: 21.

88 papers.

Sort:

By area:

  1. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  4. 2025Improving Video Generation with Human Feedbackcited by 267
  5. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  6. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  7. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  8. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  9. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  10. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  11. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  12. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
38 more
  1. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  2. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  3. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?cited by 43
  4. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  5. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  6. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  7. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  8. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  9. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 27
  10. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  11. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 24
  12. 2024Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputscited by 22
  13. 2025Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedbackcited by 22
  14. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  15. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  16. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  17. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  18. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  19. 2025CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalizationcited by 15
  20. 2025FactSelfCheck: Fact-Level Black-Box Hallucination Detection for LLMscited by 14
  21. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  22. 2025Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critiquecited by 13
  23. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  24. 2025RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiquescited by 12
  25. 2025LLMs can be easily Confused by Instructional Distractionscited by 12
  26. 2025MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identificationcited by 12
  27. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 12
  28. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 12
  29. 2025Evaluating System 1 vs. 2 Reasoning Approaches for Zero-Shot Time Series Forecasting: A Benchmark and Insightscited by 11
  30. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  31. 2024SCITAT: A Question Answering Benchmark for Scientific Tables and Text Covering Diverse Reasoning Typescited by 10
  32. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  33. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 9
  34. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9
  35. 2025VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generationcited by 8
  36. 2024PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQLcited by 7
  37. 2025RFCAudit: An LLM Agent for Functional Bug Detection in Network Protocolscited by 7
  38. 2025Training Language Model to Critique for Better Refinementcited by 7

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
  2. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models
  3. 2025Self-Evolving Critique Abilities in Large Language Models
  4. 2025Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
  5. 2025Stepwise Think-Critique: Interleaved Reasoning and Self-Critique in a Single LLM
  6. 2025Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
  7. 2024Are Your LLMs Capable of Stable Reasoning?
  8. 2023MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models