mlsanityData to 7 Oct 2026

On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks

arXiv 2402.08115 ↗First posted 12 Feb 2024cs.AIcited by 137

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

There has been considerable divergence of opinion on the reasoning abilities of Large Language Models (LLMs). While the initial optimism that reasoning might emerge automatically with scale has been tempered thanks to a slew of counterexamples--ranging from multiplication to simple planning--there persists a wide spread belief that LLMs can self-critique and improve their own solutions in an iterative fashion. This belief seemingly rests on the assumption that verification of correctness should be easier than generation--a rather classical argument from computational complexity--which should be irrelevant to LLMs to the extent that what they are doing is approximate retrieval. In this paper, we set out to systematically investigate the effectiveness of iterative prompting in the context of reasoning and planning. We present a principled empirical study of the performance of GPT-4 in three domains: Game of 24, Graph Coloring, and STRIPS planning. We experiment both with the model critiquing its own answers and with an external correct reasoner verifying proposed solutions. In each case, we analyze whether the content of criticisms actually affects bottom line performance, and whether we can ablate elements of the augmented system without losing performance. We observe significant performance collapse with self-critique and significant performance gains with sound external verification. We also note that merely re-prompting with a sound verifier maintains most of the benefits of more involved setups.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 28 of its 36 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

28 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yet3 Oct
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May
  3. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2023Reasoning with Language Model is Planning with World Modelcited by 846
13 more
  1. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  2. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  3. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  4. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  5. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  6. 2022Generating Sequences by Learning to Self-Correctcited by 271
  7. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  8. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  9. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  10. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  11. 2023Data Contamination Through the Lens of Timecited by 39
  12. 2023Exploring Distributional Shifts in Large Language Models for Code Analysiscited by 14
  13. 2023KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrievalcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 61; 2026: 49.

137 papers.

Sort:

By area:

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  2. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  3. 2024Stream of Search (SoS): Learning to Search in Languagecited by 141
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 123
  6. 2025Confidence Improves Self-Consistency in LLMscited by 111
  7. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  8. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  9. 2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 95
  10. 2024Formal Mathematical Reasoning: A New Frontier in AIcited by 80
  11. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
  12. 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
38 more
  1. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?cited by 43
  2. 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraintscited by 43
  3. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  4. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoningcited by 40
  5. 2024Robust Planning with LLM-Modulo Framework: Case Study in Travel Planningcited by 34
  6. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  7. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  8. 2025Entropy-based Exploration Conduction for Multi-step Reasoningcited by 28
  9. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  10. 2025Prover Agent: An Agent-Based Framework for Formal Mathematical Proofscited by 28
  11. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  12. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 24
  13. 2025Knowing You Don't Know: Learning When to Continue Search in Multi-round RAG through Self-Practicingcited by 24
  14. 2025Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMscited by 22
  15. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  16. 2024Revealing the Barriers of Language Agents in Planningcited by 20
  17. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 19
  18. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 18
  19. 2024On the Brittle Foundations of ReAct Prompting for Agentic Large Language Modelscited by 17
  20. 2024Planning in Strawberry Fields: Evaluating and Improving the Planning and Scheduling Capabilities of LRM o1cited by 17
  21. 2025A Survey on Large Language Models for Automated Planningcited by 17
  22. 2025Learning to Generate Unit Tests for Automated Debuggingcited by 16
  23. 2025Teaching LLMs to Plan: Logical Chain-of-Thought Instruction Tuning for Symbolic Planningcited by 15
  24. 2025VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failurescited by 14
  25. 2026DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generationcited by 14
  26. 2024RLSF: Fine-tuning LLMs via Symbolic Feedbackcited by 13
  27. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMscited by 12
  28. 2025RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiquescited by 12
  29. 2025Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verificationcited by 12
  30. 2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
  31. 2025When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifierscited by 11
  32. 2025LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actionscited by 10
  33. 2025Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloringcited by 10
  34. 2025VerifiAgent: a Unified Verification Agent in Language Model Reasoningcited by 10
  35. 2025Zero-Shot Verification-guided Chain of Thoughtscited by 9
  36. 2025Towards Enhanced Immersion and Agency for LLM-based Interactive Dramacited by 9
  37. 2025Verification-Aware Planning for Multi-Agent Systemscited by 9
  38. 2025Reasoning with Graphs: Structuring Implicit Knowledge to Enhance LLMs Reasoningcited by 8

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Capabilities of Large Language Models on Dynamic Tasks
  2. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems
  3. 2025On the Limits of Innate Planning in Large Language Models
  4. 2024A NotSo Simple Way to Beat Simple Bench
  5. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks
  6. 2026SokoBench: Evaluating Long-Horizon Planning and Reasoning in Large Language Models
  7. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
  8. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insights