Large Language Model Reasoning Failures
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Large Language Models (LLMs) have exhibited remarkable reasoning capabilities, achieving impressive results across a wide range of tasks. Despite these advances, significant reasoning failures persist, occurring even in seemingly simple scenarios. To systematically understand and address these shortcomings, we present the first comprehensive survey dedicated to reasoning failures in LLMs. We introduce a novel categorization framework that distinguishes reasoning into embodied and non-embodied types, with the latter further subdivided into informal (intuitive) and formal (logical) reasoning. In parallel, we classify reasoning failures along a complementary axis into three types: fundamental failures intrinsic to LLM architectures that broadly affect downstream tasks; application-specific limitations that manifest in particular domains; and robustness issues characterized by inconsistent performance across minor variations. For each reasoning failure, we provide a clear definition, analyze existing studies, explore root causes, and present mitigation strategies. By unifying fragmented research efforts, our survey provides a structured perspective on systemic weaknesses in LLM reasoning, offering valuable insights and guiding future research towards building stronger, more reliable, and robust reasoning capabilities. We additionally release a comprehensive collection of research works on LLM reasoning failures, as a GitHub repository at https://github.com/Peiyang-Song/Awesome-LLM-Reasoning-Failures, to provide an easy entry point to this area.
Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 318 of its 472 references that are in the map. First posted on arXiv 5 Feb 2026.
Builds on
318 of 472 references.
Closest ancestors · 32
Same direction, newest first
- 2025How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns30 Dec
- 2025Structured Moral Reasoning in Language Models: A Value-Grounded Evaluation Framework17 Jun
- 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Study14 Apr
- 2025Logical Reasoning in Large Language Models: A Survey13 Feb
- 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models16 Jan
- 2024Inducing Human-like Biases in Moral Reasoning Language Models23 Nov
- 2024RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning21 Oct
- 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data19 Sep
- 2024Multi-Step Reasoning with Large Language Models, a Survey16 Jul
- 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasoners16 Jun
- 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Models16 Jun
- 2024Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responses9 Jun
20 more
- 2024Easy Problems That LLMs Get Wrong30 May
- 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in29 Apr
- 2024GreedLlama: Performance of Financial Value-Aligned Large Language Models in Moral Reasoning3 Apr
- 2024Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts17 Mar
- 2024Exploring the Limitations of Large Language Models in Compositional Relation Reasoning5 Mar
- 2024Understanding and Patching Compositional Reasoning in LLMs22 Feb
- 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs18 Feb
- 2024Assessing the Reasoning Capabilities of LLMs in the context of Evidence-based Claim Verification16 Feb
- 2023A Survey of Reasoning with Foundation Models17 Dec
- 2023Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models6 Dec
- 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models25 Oct
- 2023Probing the Moral Development of Large Language Models through Defining Issues Test23 Sep
- 2023Evaluating Large Language Models with NeuBAROCO: Syllogistic Reasoning Ability and Human-like Biases21 Jun
- 2023Large Language Models Are Not Strong Abstract Reasoners31 May
- 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models24 May
- 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
- 2023Natural Language Reasoning, A Survey26 Mar
- 2022Towards Reasoning in Large Language Models: A Survey20 Dec
- 2022Mind's Eye: Grounded Language Model Reasoning through Simulation11 Oct
- 2022On the Paradox of Learning to Reason from Data23 May
Other references · 286
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
38 more
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2024Mixtral of Expertscited by 1,785
- 2025s1: Simple test-time scalingcited by 1,311
- 2024OpenAI o1 System Cardcited by 1,220
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
- 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
- 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
- 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
- 2025Humanity's Last Examcited by 467
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
- 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
- 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
- 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
- 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
- 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
- 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
- 2023Provable Robust Watermarking for AI-Generated Textcited by 259
- 2022Reasoning with Language Model Prompting: A Surveycited by 251
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
- 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
- 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
- 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
- 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
- 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
- 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
- 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
- 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
- 2022Large Language Models with Controllable Working Memorycited by 169
50 most cited of 286 All 286 (JSON)
Built on by
Citing papers per year: 2026: 20.
20 papers.
Sort:
By area:
- 2026Recursive Multi-Agent Systemscited by 10
- 2026MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Modelscited by 6
- 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 5
- 2026Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrencecited by 3
- 2026Interactive Evaluation Requires a Design Sciencecited by 3
- 2026MLLM-based Textual Explanations for Face Comparisoncited by 2
- 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 2
- 2026Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoningcited by 1
- 2026Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behaviorcited by 1
- 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Modelscited by 1
- 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learningcited by 1
- 2026Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problemcited by 0
8 more
- 2026Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflowscited by 0
- 2026FitText: Evolving Agent Tool Ecologies via Memetic Retrievalcited by 0
- 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
- 2026Implicit Reasoning Steering via Concept Chainingcited by 0
- 2026Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Modelscited by 0
- 2026Thought-Level Beam Search for Reasoningcited by 0
- 2026Position: Reasoning is a Learnable Rule-Based Processcited by 0
- 2026Walking on the DARKSIDEcited by 0
- 2026Walking on the DARKSIDE24 Aug 2026
- 2026Thought-Level Beam Search for Reasoning8 Aug 2026
- 2026Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Models2 Aug 2026
- 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning24 Jul 2026
- 2026Implicit Reasoning Steering via Concept Chaining15 Jul 2026
- 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models22 Jun 2026
- 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs16 Jun 2026
- 2026Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior10 Jun 2026
- 2026Position: Reasoning is a Learnable Rule-Based Process29 May 2026
- 2026Interactive Evaluation Requires a Design Science18 May 2026
- 2026FitText: Evolving Agent Tool Ecologies via Memetic Retrieval4 May 2026
- 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions30 Apr 2026
8 more
- 2026Recursive Multi-Agent Systems28 Apr 2026
- 2026Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflows17 Apr 2026
- 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistants28 Mar 2026
- 2026MLLM-based Textual Explanations for Face Comparison17 Mar 2026
- 2026Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning9 Mar 2026
- 2026Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problem7 Mar 2026
- 2026MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Models17 Feb 2026
- 2026Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrence11 Feb 2026
- 2026Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoningcited by 1
- 2026Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behaviorcited by 1
- 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Modelscited by 1
- 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learningcited by 1
- 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMscited by 0
- 2026Implicit Reasoning Steering via Concept Chainingcited by 0
- 2026Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Modelscited by 0
- 2026Thought-Level Beam Search for Reasoningcited by 0
- 2026Thought-Level Beam Search for Reasoning8 Aug 2026
- 2026Cognitive Demand Steering for Adaptive Meta-Reasoning in Large Language Models2 Aug 2026
- 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning24 Jul 2026
- 2026Implicit Reasoning Steering via Concept Chaining15 Jul 2026
- 2026ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models22 Jun 2026
- 2026Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs16 Jun 2026
- 2026Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior10 Jun 2026
- 2026Learning When to Sample: Confidence-Aware Selective Sampling for Efficient Chain-of-Thought Reasoning9 Mar 2026
- 2026MLLM-based Textual Explanations for Face Comparisoncited by 2
- 2026MLLM-based Textual Explanations for Face Comparison17 Mar 2026
- 2026MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Modelscited by 6
- 2026Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrencecited by 3
- 2026Interactive Evaluation Requires a Design Sciencecited by 3
- 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 2
- 2026Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problemcited by 0
- 2026FitText: Evolving Agent Tool Ecologies via Memetic Retrievalcited by 0
- 2026Position: Reasoning is a Learnable Rule-Based Processcited by 0
- 2026Walking on the DARKSIDEcited by 0
- 2026Walking on the DARKSIDE24 Aug 2026
- 2026Position: Reasoning is a Learnable Rule-Based Process29 May 2026
- 2026Interactive Evaluation Requires a Design Science18 May 2026
- 2026FitText: Evolving Agent Tool Ecologies via Memetic Retrieval4 May 2026
- 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actions30 Apr 2026
- 2026Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problem7 Mar 2026
- 2026MadEvolve: Evolutionary Optimization of Cosmological Algorithms with Large Language Models17 Feb 2026
- 2026Predictive Associative Memory: Retrieval Beyond Similarity Through Temporal Co-occurrence11 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
- 2026Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
- 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes
- 2026Opening the Black Box: A Survey on the Mechanisms of Multi-Step Reasoning in Large Language Models
- 2025A Survey of Scaling in Large Language Model Reasoning
- 2025A Modular Dataset to Demonstrate LLM Abstraction Capability
- 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models