mlsanityData to 7 Oct 2026

LLM Critics Help Catch LLM Bugs

arXiv 2407.00215 ↗First posted 28 Jun 2024cs.SE cs.LGcited by 138

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is fundamentally limited by the capacity of humans to correctly evaluate model output. To improve human evaluation ability and overcome that limitation this work trains "critic" models that help humans to more accurately evaluate model-written code. These critics are themselves LLMs trained with RLHF to write natural language feedback highlighting problems in code from real-world assistant tasks. On code containing naturally occurring LLM errors model-written critiques are preferred over human critiques in 63% of cases, and human evaluation finds that models catch more bugs than human contractors paid for code review. We further confirm that our fine-tuned LLM critics can successfully identify hundreds of errors in ChatGPT training data rated as "flawless", even though the majority of those tasks are non-code tasks and thus out-of-distribution for the critic model. Critics can have limitations of their own, including hallucinated bugs that could mislead humans into making mistakes they might have otherwise avoided, but human-machine teams of critics and contractors catch similar numbers of bugs to LLM critics while hallucinating less than LLMs alone.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 21 of its 28 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

21 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Teaching Large Language Models to Self-Debug11 Apr

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  8. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2023Language Models can Solve Computer Taskscited by 371
  11. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  12. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
8 more
  1. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  2. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  3. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  4. 2022Defining and Characterizing Reward Hackingcited by 184
  5. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  6. 2022Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agentscited by 62
  7. 2022AUGER: Automatically Generating Review Comments with Pre-training Modelscited by 23
  8. 2024Self-Refinement of Language Models from External Proxy Metrics Feedbackcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 69; 2026: 32.

138 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  4. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  5. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  6. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 277
  7. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  8. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  9. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  10. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  11. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  12. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
38 more
  1. 2024Critique-out-Loud Reward Modelscited by 105
  2. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  3. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  4. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  5. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  6. 2024MALT: Improving Reasoning with Multi-Agent LLM Trainingcited by 71
  7. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  8. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  9. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  12. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoningcited by 50
  13. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  14. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  15. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  16. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigationcited by 36
  17. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  18. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generationcited by 34
  19. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  20. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  21. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  22. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  23. 2025Heimdall: test-time scaling on the generative verificationcited by 26
  24. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoningcited by 23
  25. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 22
  26. 2025Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMscited by 22
  27. 2024Learning Code Preference via Synthetic Evolutioncited by 19
  28. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  29. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  30. 2024AIGS: Generating Science from AI-Powered Automated Falsificationcited by 17
  31. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  32. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16
  33. 2025Dynamic and Generalizable Process Reward Modelingcited by 16
  34. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Modelscited by 14
  35. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  36. 2026BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarkscited by 13
  37. 2025Self-Evolving Critique Abilities in Large Language Modelscited by 12
  38. 2025RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiquescited by 12

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  2. 2025Dean of LLM Tutors: A Framework for Automated Quality Review of AI-generated Feedback
  3. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
  4. 2025Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts
  5. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
  6. 2025ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback
  7. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMs
  8. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning