mlsanityData to 7 Oct 2026

Easy Problems That LLMs Get Wrong

arXiv 2405.19616 ↗First posted 30 May 2024cs.AI cs.CL cs.LGcited by 29

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We introduce a comprehensive Linguistic Benchmark designed to evaluate the limitations of Large Language Models (LLMs) in domains such as logical reasoning, spatial intelligence, and linguistic understanding, among others. Through a series of straightforward questions, it uncovers the significant limitations of well-regarded models to perform tasks that humans manage with ease. It also highlights the potential of prompt engineering to mitigate some errors and underscores the necessity for better training methodologies. Our findings stress the importance of grounding LLMs with human reasoning and common sense, emphasising the need for human-in-the-loop for enterprise applications. We hope this work paves the way for future research to enhance the usefulness and reliability of new models.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 17 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

12 of 17 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey17 Feb
  2. 2024LLMs for Relational Reasoning: How Far are We?17 Jan

Other references · 10

  1. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  2. 2023Language Modeling Is Compressioncited by 225
  3. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  4. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 168
  5. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 134
  6. 2024Datasets for Large Language Models: A Comprehensive Surveycited by 69
  7. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  8. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
  9. 2023Limits for Learning with Language Modelscited by 17
  10. 2023Mathematics, word problems, common sense, and artificial intelligencecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 19; 2026: 3.

29 papers.

Sort:

By area:

  1. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  2. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  3. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  4. 2024To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planningcited by 21
  5. 2025Frontier LLMs Still Struggle with Simple Reasoning Taskscited by 20
  6. 2026Large Language Model Reasoning Failurescited by 20
  7. 2024The Illusion-Illusion: Vision Language Models See Illusions Where There Are Nonecited by 14
  8. 2025SPEX: Scaling Feature Interaction Explanations for LLMscited by 12
  9. 2025Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitivescited by 7
  10. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agentscited by 7
  11. 2025The Philosophical Foundations of Growing AI Like A Childcited by 6
  12. 2025Reasoning Model is Stubborn: Diagnosing Instruction Overriding in Reasoning Modelscited by 6
17 more
  1. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  2. 2025LLMs Encode How Difficult Problems Arecited by 6
  3. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  4. 2024United in Diversity? Contextual Biases in LLM-Based Predictions of the 2024 European Parliament Electionscited by 5
  5. 2024Enhancing LLM Problem Solving with REAP: Reflection, Explicit Problem Deconstruction, and Advanced Promptingcited by 4
  6. 2025Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigationcited by 4
  7. 2025Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolutioncited by 3
  8. 2025ConDABench: Interactive Evaluation of Language Models for Data Analysiscited by 3
  9. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3
  10. 2025Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoningcited by 2
  11. 2026Certainty robustness: Evaluating LLM stability under self-challenging promptscited by 2
  12. 2024Human-AI collectives produce the most accurate differential diagnosescited by 1
  13. 2025Foundational Design Principles and Patterns for Building Robust and Adaptive GenAI-Native Systemscited by 1
  14. 2025A Criminology of Machinescited by 1
  15. 2025On the Brittleness of LLMs: A Journey around Set Membershipcited by 1
  16. 2024Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?cited by 0
  17. 2025PHANTOM RECALL: When Familiar Puzzles Fool Smart Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  2. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  3. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  4. 2022Towards Reasoning in Large Language Models: A Survey
  5. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  6. 2025Reasoning Capabilities and Invariability of Large Language Models
  7. 2025DeepQuestion: Systematic Generation of Real-World Challenges for Evaluating LLMs Performance
  8. 2025Large Language Models and Mathematical Reasoning Failures