mlsanityData to 14 Oct 2026

GPT-4 Can't Reason

arXiv 2308.03762 ↗First posted 21 Jul 2023cs.CLcited by 40

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

GPT-4 was released in March 2023 to wide acclaim, marking a very substantial improvement across the board over GPT-3.5 (OpenAI's previously best model, which had powered the initial release of ChatGPT). However, despite the genuinely impressive improvement, there are good reasons to be highly skeptical of GPT-4's ability to reason. This position paper discusses the nature of reasoning; criticizes the current formulation of reasoning problems in the NLP community, as well as the way in which LLM reasoning performance is currently evaluated; introduces a small collection of 21 diverse reasoning problems; and performs a detailed qualitative evaluation of GPT-4's performance on those problems. Based on this analysis, the paper concludes that, despite its occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 24 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

6 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct
  3. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning1 May

Other references · 3

  1. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  2. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  3. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 12; 2025: 11; 2026: 4.

40 papers.

Sort:

By area:

  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  3. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  4. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  5. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
  6. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  7. 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 51
  8. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 50
  9. 2023A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoningcited by 41
  10. 2023DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacycited by 34
  11. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 31
  12. 2023SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluationcited by 27
28 more
  1. 2024Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Modelscited by 23
  2. 2023Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Modelscited by 21
  3. 2025Potemkin Understanding in Large Language Modelscited by 15
  4. 2025Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transitioncited by 14
  5. 2024Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilitiescited by 13
  6. 2024Evaluating LLM Reasoning in the Operations Research Domain with ORQAcited by 12
  7. 2025Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloringcited by 10
  8. 2023Chain-of-Thought Reasoning is a Policy Improvement Operatorcited by 8
  9. 2023How should the advent of large language models affect the practice of science?cited by 8
  10. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  11. 2025Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Modelscited by 7
  12. 2024Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilitiescited by 6
  13. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approachcited by 6
  14. 2025The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?cited by 6
  15. 2024Information Re-Organization Improves Reasoning in Large Language Modelscited by 5
  16. 2024The application of GPT-4 in grading design university students' assignment and providing feedback: An exploratory studycited by 4
  17. 2025Mathematical Computation and Reasoning Errors by Large Language Modelscited by 4
  18. 2026The unreasonable effectiveness of pattern matchingcited by 3
  19. 2023Concepts is All You Need: A More Direct Path to AGIcited by 2
  20. 2024Aligning Generalisation Between Humans and Machinescited by 2
  21. 2024A Theoretical Survey on Foundation Modelscited by 1
  22. 2024A Statistical Analysis of LLMs' Self-Evaluation Using Proverbscited by 1
  23. 2025Computational Reasoning of Large Language Modelscited by 1
  24. 2025SVAgent: AI Agent for Hardware Security Verification Assertioncited by 1
  25. 2025Causes in neuron diagrams, and testing causal reasoning in Large Language Models. A glimpse of the future of philosophy?cited by 0
  26. 2026IFDNS: An Iterative Feedback-Driven Neuro-Symbolic Method for Faithful Logical Reasoningcited by 0
  27. 2026CogBias: Measuring and Mitigating Cognitive Bias in Large Language Modelscited by 0
  28. 2026Reasoning as Pattern Matching: Shared Mechanisms in Human and LLM Everyday Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4
  2. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts
  3. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
  4. 2023Using Language Models For Knowledge Acquisition in Natural Language Reasoning Problems
  5. 2023Consistency Analysis of ChatGPT
  6. 2023How is ChatGPT's behavior changing over time?
  7. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification
  8. 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation