# Easy Problems That LLMs Get Wrong

arXiv: https://arxiv.org/abs/2405.19616 · HTML: https://mlsanity.com/paper/2405.19616/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 12 of its 17 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 12 of 12)

- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [An Empirical Study of the Non-determinism of ChatGPT in Code Generation](https://mlsanity.com/paper/2308.02828/) (2023)
- [Datasets for Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2402.18041/) (2024)
- [Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence](https://mlsanity.com/paper/2402.09880/) (2024)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models](https://mlsanity.com/paper/2404.06209/) (2024)
- [LLMs for Relational Reasoning: How Far are We?](https://mlsanity.com/paper/2401.09042/) (2024)
- [Limits for Learning with Language Models](https://mlsanity.com/paper/2306.12213/) (2023)
- [Mathematics, word problems, common sense, and artificial intelligence](https://mlsanity.com/paper/2301.09723/) (2023)

## Built on by (most cited first, 30 of 30)

- [Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.24726/) (2025)
- [Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina](https://mlsanity.com/paper/2410.19599/) (2024)
- [PDE-Controller: LLMs for Autoformalization and Reasoning of PDEs](https://mlsanity.com/paper/2502.00963/) (2025)
- [To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning](https://mlsanity.com/paper/2410.16456/) (2024)
- [Frontier LLMs Still Struggle with Simple Reasoning Tasks](https://mlsanity.com/paper/2507.07313/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [The Illusion-Illusion: Vision Language Models See Illusions Where There Are None](https://mlsanity.com/paper/2412.18613/) (2024)
- [SPEX: Scaling Feature Interaction Explanations for LLMs](https://mlsanity.com/paper/2502.13870/) (2025)
- [When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents](https://mlsanity.com/paper/2507.09329/) (2025)
- [Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives](https://mlsanity.com/paper/2502.04358/) (2025)
- [LLMs Encode How Difficult Problems Are](https://mlsanity.com/paper/2510.18147/) (2025)
- [The Philosophical Foundations of Growing AI Like A Child](https://mlsanity.com/paper/2502.10742/) (2025)
- [Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Models](https://mlsanity.com/paper/2505.17225/) (2025)
- [Beyond Statistical Learning: Exact Learning Is Essential for General Intelligence](https://mlsanity.com/paper/2506.23908/) (2025)
- [Generative AI collective behavior needs an interactionist paradigm](https://mlsanity.com/paper/2601.10567/) (2026)
- [United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Elections](https://mlsanity.com/paper/2409.09045/) (2024)
- [Enhancing LLM Problem Solving with REAP: Reflection, Explicit Problem Deconstruction, and Advanced Prompting](https://mlsanity.com/paper/2409.09415/) (2024)
- [Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigation](https://mlsanity.com/paper/2504.08806/) (2025)
- [Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution](https://mlsanity.com/paper/2508.07111/) (2025)
- [ConDABench: Interactive Evaluation of Language Models for Data Analysis](https://mlsanity.com/paper/2510.13835/) (2025)
- [Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings](https://mlsanity.com/paper/2512.14917/) (2025)
- [Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoning](https://mlsanity.com/paper/2508.10057/) (2025)
- [Certainty robustness: Evaluating LLM stability under self-challenging prompts](https://mlsanity.com/paper/2603.03330/) (2026)
- [Human-AI collectives produce the most accurate differential diagnoses](https://mlsanity.com/paper/2406.14981/) (2024)
- [Foundational Design Principles and Patterns for Building Robust and Adaptive GenAI-Native Systems](https://mlsanity.com/paper/2508.15411/) (2025)
- [A Criminology of Machines](https://mlsanity.com/paper/2511.02895/) (2025)
- [On the Brittleness of LLMs: A Journey around Set Membership](https://mlsanity.com/paper/2511.12728/) (2025)
- [Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?](https://mlsanity.com/paper/2410.16285/) (2024)
- [PHANTOM RECALL: When Familiar Puzzles Fool Smart Models](https://mlsanity.com/paper/2510.11812/) (2025)
- [From Benchmark to Bench: Can Agents Survive Real-World Drug Discovery?](https://mlsanity.com/paper/2610.06411/) (2026)

## Abstract (arXiv)

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others. Through a series of straightforward questions, it uncovers the significant limitations of well-regarded models to perform tasks that humans manage with ease. It also highlights the potential of prompt engineering to mitigate some errors and underscores the necessity for better training methodologies. Our findings stress the importance of grounding LLMs with human reasoning and common sense, emphasising the need for human-in-the-loop for enterprise applications. We hope this work paves the way for future research to enhance the usefulness and reliability of new models.