# GPT-4 Can't Reason

arXiv: https://arxiv.org/abs/2308.03762 · HTML: https://mlsanity.com/paper/2308.03762/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 6 of its 24 references that are in the map.
- First posted on arXiv 21 Jul 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 6 of 6)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4](https://mlsanity.com/paper/2304.03439/) (2023)
- [MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning](https://mlsanity.com/paper/2205.00445/) (2022)

## Built on by (most cited first, 30 of 40)

- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems](https://mlsanity.com/paper/2310.12397/) (2023)
- [MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation](https://mlsanity.com/paper/2312.17080/) (2023)
- [Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing](https://mlsanity.com/paper/2309.05898/) (2023)
- [Transformers Provably Solve Parity Efficiently with Chain of Thought](https://mlsanity.com/paper/2410.08633/) (2024)
- [A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning](https://mlsanity.com/paper/2311.07954/) (2023)
- [DetermLR: Augmenting LLM-based Logical Reasoning from Indeterminacy to Determinacy](https://mlsanity.com/paper/2310.18659/) (2023)
- [Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning](https://mlsanity.com/paper/2502.04381/) (2025)
- [SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation](https://mlsanity.com/paper/2310.00074/) (2023)
- [Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models](https://mlsanity.com/paper/2409.17539/) (2024)
- [Memory Injections: Correcting Multi-Hop Reasoning Failures during Inference in Transformer-Based Language Models](https://mlsanity.com/paper/2309.05605/) (2023)
- [Potemkin Understanding in Large Language Models](https://mlsanity.com/paper/2506.21521/) (2025)
- [Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transition](https://mlsanity.com/paper/2504.03930/) (2025)
- [Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilities](https://mlsanity.com/paper/2409.07638/) (2024)
- [Evaluating LLM Reasoning in the Operations Research Domain with ORQA](https://mlsanity.com/paper/2412.17874/) (2024)
- [Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring](https://mlsanity.com/paper/2502.07087/) (2025)
- [Chain-of-Thought Reasoning is a Policy Improvement Operator](https://mlsanity.com/paper/2309.08589/) (2023)
- [How should the advent of large language models affect the practice of science?](https://mlsanity.com/paper/2312.03759/) (2023)
- [Conditions for Length Generalization in Learning Reasoning Skills](https://mlsanity.com/paper/2311.16173/) (2023)
- [Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models](https://mlsanity.com/paper/2503.06263/) (2025)
- [Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilities](https://mlsanity.com/paper/2405.11841/) (2024)
- [Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach](https://mlsanity.com/paper/2410.21779/) (2024)
- [The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?](https://mlsanity.com/paper/2502.17535/) (2025)
- [Information Re-Organization Improves Reasoning in Large Language Models](https://mlsanity.com/paper/2404.13985/) (2024)
- [The application of GPT-4 in grading design university students' assignment and providing feedback: An exploratory study](https://mlsanity.com/paper/2409.17698/) (2024)
- [Mathematical Computation and Reasoning Errors by Large Language Models](https://mlsanity.com/paper/2508.09932/) (2025)
- [The unreasonable effectiveness of pattern matching](https://mlsanity.com/paper/2601.11432/) (2026)

## Abstract (arXiv)

GPT-4 was released in March 2023 to wide acclaim, marking a very substantial improvement across the board over GPT-3.5 (OpenAI's previously best model, which had powered the initial release of ChatGPT). However, despite the genuinely impressive improvement, there are good reasons to be highly skeptical of GPT-4's ability to reason. This position paper discusses the nature of reasoning; criticizes the current formulation of reasoning problems in the NLP community, as well as the way in which LLM reasoning performance is currently evaluated; introduces a small collection of 21 diverse reasoning problems; and performs a detailed qualitative evaluation of GPT-4's performance on those problems. Based on this analysis, the paper concludes that, despite its occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.