mlsanityData to 7 Oct 2026

Large Language Model Reasoning Failures

arXiv 2602.06176 ↗First posted 5 Feb 2026cs.AI cs.CL cs.LGcited by 20

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios. To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in LLMs. We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into informal (intuitive) and formal (logical) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to LLM architectures that broadly affect downstream tasks; application-specific limitations that manifest in particular domains; and robustness issues characterized by inconsistent performance across minor variations. For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes, and present mitigation strategies. By unifying fragmented research efforts, our survey provides a structured perspective on systemic weaknesses in LLM reasoning, offering valuable insights and guiding future research towards building stronger, more reliable, and robust reasoning capabilities. We additionally release a comprehensive collection of research works on LLM reasoning failures, as a GitHub repository at https://github.com/Peiyang-Song/Awesome-LLM-Reasoning-Failures, to provide an easy entry point to this area.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 318 of its 472 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

318 of 472 references.

Closest ancestors · 32

Same direction, newest first

  1. 2025How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns30 Dec
  2. 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework17 Jun
  3. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Study14 Apr
  4. 2025Logical Reasoning in Large Language Models: A Survey13 Feb
  5. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models16 Jan
  6. 2024Inducing Human-like Biases in Moral Reasoning Language Models23 Nov
  7. 2024RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning21 Oct
  8. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data19 Sep
  9. 2024Multi-Step Reasoning with Large Language Models, a Survey16 Jul
  10. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners16 Jun
  11. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models16 Jun
  12. 2024Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responses9 Jun
20 more
  1. 2024Easy Problems That LLMs Get Wrong30 May
  2. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in29 Apr
  3. 2024GreedLlama: Performance of Financial Value-Aligned Large Language Models in Moral Reasoning3 Apr
  4. 2024Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts17 Mar
  5. 2024Exploring the Limitations of Large Language Models in Compositional Relation Reasoning5 Mar
  6. 2024Understanding and Patching Compositional Reasoning in LLMs22 Feb
  7. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs18 Feb
  8. 2024Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification16 Feb
  9. 2023A Survey of Reasoning with Foundation Models17 Dec
  10. 2023Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models6 Dec
  11. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models25 Oct
  12. 2023Probing the Moral Development of Large Language Models through Defining Issues Test23 Sep
  13. 2023Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases21 Jun
  14. 2023Large Language Models Are Not Strong Abstract Reasoners31 May
  15. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models24 May
  16. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  17. 2023Natural Language Reasoning, A Survey26 Mar
  18. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  19. 2022Mind's Eye: Grounded Language Model Reasoning through Simulation11 Oct
  20. 2022On the Paradox of Learning to Reason from Data23 May

Other references · 286

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2024Mixtral of Expertscited by 1,785
  5. 2025s1: Simple test-time scalingcited by 1,311
  6. 2024OpenAI o1 System Cardcited by 1,220
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2023Steering Language Models With Activation Engineeringcited by 824
  13. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  14. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  15. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  16. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  17. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  18. 2025Humanity's Last Examcited by 467
  19. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  20. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  21. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  22. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  23. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
  24. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  25. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  26. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  27. 2023Provable Robust Watermarking for AI-Generated Textcited by 259
  28. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  29. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  30. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  31. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  32. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  33. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  34. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  35. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  36. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  37. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  38. 2022Large Language Models with Controllable Working Memorycited by 169

50 most cited of 286 All 286 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 20.

20 papers.

Sort:

By area:

  1. 2026Recursive Multi-Agent Systemscited by 10
  2. 2026MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Modelscited by 6
  3. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 5
  4. 2026Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrencecited by 3
  5. 2026Interactive Evaluation Requires a Design Sciencecited by 3
  6. 2026MLLM-based Textual Explanations for Face Comparisoncited by 2
  7. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 2
  8. 2026Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoningcited by 1
  9. 2026Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behaviorcited by 1
  10. 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Modelscited by 1
  11. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learningcited by 1
  12. 2026Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problemcited by 0
8 more
  1. 2026Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflowscited by 0
  2. 2026FitText: Evolving Agent Tool Ecologies via Memetic Retrievalcited by 0
  3. 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
  4. 2026Implicit Reasoning Steering via Concept Chainingcited by 0
  5. 2026Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Modelscited by 0
  6. 2026Thought-Level Beam Search for Reasoningcited by 0
  7. 2026Position: Reasoning is a Learnable Rule-Based Processcited by 0
  8. 2026Walking on the DARKSIDEcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  2. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  3. 2026Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
  4. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
  5. 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
  6. 2025A Survey of Scaling in Large Language Model Reasoning
  7. 2025A Modular Dataset to Demonstrate LLM Abstraction Capability
  8. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models