mlsanityData to 7 Oct 2026

LLMs cannot find reasoning errors, but can correct them given the error location

arXiv 2311.08516 ↗First posted 14 Nov 2023cs.AI cs.CL cs.LGcited by 85

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

While self-correction has shown promise in improving LLM outputs in terms of style and quality (e.g. Chen et al., 2023b; Madaan et al., 2023), recent attempts to self-correct logical or reasoning errors often cause correct answers to become incorrect, resulting in worse performances overall (Huang et al., 2023). In this paper, we show that poor self-correction performance stems from LLMs' inability to find logical mistakes, rather than their ability to correct a known mistake. Firstly, we benchmark several state-of-the-art LLMs on their mistake-finding ability and demonstrate that they generally struggle with the task, even in highly objective, unambiguous cases. Secondly, we test the correction abilities of LLMs -- separately from mistake finding -- using a backtracking setup that feeds ground truth mistake location information to the model. We show that this boosts downstream task performance across our 5 reasoning tasks, indicating that LLMs' correction abilities are robust. Finally, we show that it is possible to obtain mistake location information without ground truth labels or in-domain training data. We train a small classifier with out-of-domain data, which exhibits stronger mistake-finding performance than prompting a large model. We release our dataset of LLM-generated logical mistakes, BIG-Bench Mistake, to enable further research into locating LLM reasoning mistakes.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 23 of its 39 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

23 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  2. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies6 Aug
  3. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning1 Aug
  4. 2023Self-Evaluation Guided Beam Search for Reasoning1 May
  5. 2023REFINER: Reasoning Feedback on Intermediate Representations4 Apr
  6. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar
  7. 2022Large Language Models Can Self-Improve20 Oct

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2023Language Models can Solve Computer Taskscited by 371
  12. 2022CodeT: Code Generation with Generated Testscited by 369
4 more
  1. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  2. 2022Generating Sequences by Learning to Self-Correctcited by 271
  3. 2020Graph-based, Self-Supervised Program Repair from Diagnostic Feedbackcited by 93
  4. 2021Think about it! Improving defeasible reasoning by first modeling the question scenariocited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 41; 2025: 34; 2026: 8.

85 papers.

Sort:

By area:

  1. 2025Gemma 3 Technical Reportcited by 1,686
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  4. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  5. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  6. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  7. 2025BIG-Bench Extra Hardcited by 123
  8. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  9. 2025Confidence Improves Self-Consistency in LLMscited by 111
  10. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  11. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 74
  12. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
38 more
  1. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  2. 2025Self-rewarding correction for mathematical reasoningcited by 67
  3. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 62
  4. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  5. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  6. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  7. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  8. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  9. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  10. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  11. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?cited by 43
  12. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 40
  13. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 38
  14. 2024Are Large Language Models Good Prompt Optimizers?cited by 37
  15. 2025S2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learningcited by 37
  16. 2024Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphscited by 32
  17. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  18. 2024Weak-to-Strong Reasoningcited by 32
  19. 2024Backtracking Improves Generation Safetycited by 31
  20. 2025MINERVA: Evaluating Complex Video Reasoningcited by 31
  21. 2024MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMscited by 30
  22. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  23. 2024LLMs can Find Mathematical Reasoning Mistakes by Pedagogical Chain-of-Thoughtcited by 28
  24. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  25. 2025Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysiscited by 26
  26. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 24
  27. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 23
  28. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  29. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21
  30. 2024The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?cited by 20
  31. 2024Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?cited by 18
  32. 2023Meta Prompting for AI Systemscited by 16
  33. 2024DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanismscited by 16
  34. 2024Mirror-Consistency: Harnessing Inconsistency in Majority Votingcited by 16
  35. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  36. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  37. 2025RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiquescited by 12
  38. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 12

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoning
  2. 2023Learning From Mistakes Makes LLM Better Reasoner
  3. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-context
  4. 2025Unraveling Misinformation Propagation in LLM Reasoning
  5. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning
  6. 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
  7. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correction
  8. 2026Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs