mlsanityData to 14 Oct 2026

Large Language Models Are Not Strong Abstract Reasoners

arXiv 2305.19555 ↗First posted 31 May 2023cs.CL cs.LGcited by 53

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large Language Models have shown tremendous performance on a large variety of natural language processing tasks, ranging from text comprehension to common sense reasoning. However, the mechanisms responsible for this success remain opaque, and it is unclear whether LLMs can achieve human-like cognitive capabilities or whether these models are still fundamentally circumscribed. Abstract reasoning is a fundamental task for cognition, consisting of finding and applying a general pattern from few data. Evaluating deep neural architectures on this task could give insight into their potential limitations regarding reasoning and their broad generalisation abilities, yet this is currently an under-explored area. In this paper, we introduce a new benchmark for evaluating language models beyond memorization on abstract reasoning tasks. We perform extensive evaluations of state-of-the-art LLMs, showing that they currently achieve very limited performance in contrast with other natural language tasks, even when applying techniques that have been shown to improve performance on other NLP tasks. We argue that guiding LLM generation to follow causal paths could help improve the generalisation and reasoning abilities of LLMs.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 43 of its 80 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

43 of 80 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022Language Models as Inductive Reasoners21 Dec
  4. 2022Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation28 Jul
  5. 2022AbductionRules: Training Transformers to Explain Unexpected Inputs23 Mar
  6. 2022MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning1 Mar

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
25 more
  1. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  6. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  8. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  9. 2020Transformers as Soft Reasoners over Languagecited by 270
  10. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  11. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  12. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  13. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  14. 2018Measuring abstract reasoning in neural networkscited by 166
  15. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  16. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 106
  17. 2020Thinking Fast and Slow in AIcited by 70
  18. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  19. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  20. 2021Communicating Natural Programs to Humans and Machinescited by 35
  21. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  22. 2022Can Foundation Models Talk Causality?cited by 30
  23. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  24. 2022ROCK: Causal Inference Principles for Reasoning about Commonsense Causalitycited by 16
  25. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 24; 2026: 12.

53 papers.

Sort:

By area:

  1. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
  2. 2024Efficient Causal Graph Discovery Using Large Language Modelscited by 53
  3. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  4. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  5. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 36
  6. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  7. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  8. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  9. 2026Large Language Model Reasoning Failurescited by 21
  10. 2024MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Modelscited by 17
  11. 2024ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understandingcited by 13
  12. 2024Are LLMs Good Zero-Shot Fallacy Classifiers?cited by 12
38 more
  1. 2024Towards Learning Abductive Reasoning using VSA Distributed Representationscited by 9
  2. 2024Can Large Language Models generalize analogy solving like children can?cited by 9
  3. 2025MuaLLM: A Multimodal Large Language Model Agent for Circuit Design Assistance with Hybrid Contextual Retrieval-Augmented Generationcited by 8
  4. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
  5. 2024Counterfactual Causal Inference in Natural Language with Large Language Modelscited by 7
  6. 2024Creativity in AI: Progresses and Challengescited by 7
  7. 2024Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoningcited by 7
  8. 2024Learning by Analogy: Enhancing Few-Shot Prompting for Math Word Problem Solving with Computational Graph-Based Retrievalcited by 5
  9. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5
  10. 2025Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priorscited by 5
  11. 2026A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilitiescited by 5
  12. 2025Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Modelscited by 4
  13. 2025AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinkingcited by 4
  14. 2024Critical Questions Generation: Motivation and Challengescited by 3
  15. 2025Exploring the Hidden Reasoning Process of Large Language Models by Misleading Themcited by 3
  16. 2025Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedentscited by 3
  17. 2025NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modificationcited by 3
  18. 2025Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategiescited by 3
  19. 2024Inductive Linguistic Reasoning with Large Language Modelscited by 2
  20. 2025On the Reasoning Capacity of AI Models and How to Quantify Itcited by 2
  21. 2025Large Language Models as Common-Sense Heuristicscited by 2
  22. 2025SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Modelscited by 2
  23. 2025Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoningcited by 2
  24. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  25. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoningcited by 2
  26. 2025Dynamic Reinforcement Learning for Actorscited by 1
  27. 2025Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Modelscited by 1
  28. 2025Adaptive Multi-Agent Reasoning via Automated Workflow Generationcited by 1
  29. 2025Toward a Metrology for Artificial Intelligence: Hidden-Rule Environments and Reinforcement Learningcited by 1
  30. 2025Designing AI-Resilient Assessments Using Interconnected Problems: A Theoretically Grounded and Empirically Validated Frameworkcited by 1
  31. 2025Causal Cartographer: From Mapping to Reasoning Over Counterfactual Worldscited by 0
  32. 2025Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generationcited by 0
  33. 2026BACH-V: Bridging Abstract and Concrete Human-Values in Large Language Modelscited by 0
  34. 2026Lifted Relational Probabilistic Inference via Implicit Learningcited by 0
  35. 2026Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperformcited by 0
  36. 2026Position: Hippocampal Explicit Memory Is the Cornerstone for AGIcited by 0
  37. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  38. 2026DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Modelscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  2. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
  3. 2022Towards Reasoning in Large Language Models: A Survey
  4. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  5. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
  6. 2025Towards Reasoning Ability of Small Language Models
  7. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMs
  8. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitations