mlsanityData to 7 Oct 2026

Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

arXiv 2403.11793 ↗First posted 18 Mar 2024cs.CL cs.AI cs.ETcited by 32

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The existing methods for evaluating the inference abilities of Large Language Models (LLMs) have been predominantly results-centric, making it challenging to assess the inference process comprehensively. We introduce a novel approach using the Abstraction and Reasoning Corpus (ARC) benchmark to evaluate the inference and contextual understanding abilities of LLMs in a process-centric manner, focusing on three key components from the Language of Thought Hypothesis (LoTH): Logical Coherence, Compositionality, and Productivity. Our carefully designed experiments reveal that while LLMs demonstrate some inference capabilities, they still significantly lag behind human-level reasoning in these three aspects. The main contribution of this paper lies in introducing the LoTH perspective, which provides a method for evaluating the reasoning process that conventional results-oriented approaches fail to capture, thereby offering new insights into the development of human-level reasoning in artificial intelligence systems.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 59 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

38 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Understanding and Patching Compositional Reasoning in LLMs22 Feb
  2. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement12 Oct
  3. 2023Hypothesis Search: Inductive Reasoning with Language Models11 Sep
  4. 2022Towards Reasoning in Large Language Models: A Survey20 Dec

Other references · 34

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  5. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  6. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  7. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  8. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  11. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  12. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
22 more
  1. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  2. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  3. 2023Large Language Models as General Pattern Machinescited by 196
  4. 2018Measuring abstract reasoning in neural networkscited by 165
  5. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  6. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  7. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  8. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 81
  9. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  10. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  11. 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
  12. 2019Learning to Make Analogies by Contrasting Abstract Relational Structurecited by 53
  13. 2020Learning Representations that Support Extrapolationcited by 36
  14. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33
  15. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  16. 2022Graphs, Constraints, and Search for the Abstraction and Reasoning Corpuscited by 32
  17. 2020Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoningcited by 18
  18. 2023AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graphcited by 17
  19. 2019Do Multi-hop Readers Dream of Reasoning Chains?cited by 14
  20. 2024AbsInstruct: Eliciting Abstraction Ability from LLMs through Explanation Tuning with Plausibility Estimationcited by 14
  21. 2023Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformercited by 12
  22. 2021Phy-Q as a measure for physical reasoning intelligencecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 15; 2026: 8.

32 papers.

Sort:

By area:

  1. 2024Combining Induction and Transduction for Abstract Reasoningcited by 53
  2. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  3. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 29
  4. 2024HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesiscited by 14
  5. 2025From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMscited by 12
  6. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 8
  7. 2024ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learningcited by 7
  8. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoningcited by 5
  9. 2024Program Synthesis using Inductive Logic Programming for the Abstraction and Reasoning Corpuscited by 4
  10. 2025UserCentrix: An Agentic Memory-augmented AI Framework for Smart Spacescited by 4
  11. 2025Reasoning Effort and Problem Complexity: A Scaling Analysis in LLMscited by 3
  12. 2024Enhancing Analogical Reasoning in the Abstraction and Reasoning Corpus via Model-Based RLcited by 2
20 more
  1. 2024Irrelevant Alternatives Bias Large Language Model Hiring Decisionscited by 2
  2. 2024Abductive Symbolic Solver on Abstraction and Reasoning Corpuscited by 2
  3. 2025From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmarkcited by 2
  4. 2025From Sound to Sight: Towards AI-authored Music Videoscited by 2
  5. 2025Plug-and-Play Dramaturge: A Divide-and-Conquer Approach for Iterative Narrative Script Refinement via Collaborative LLM Agentscited by 2
  6. 2026Slots, Transitions, Loops: Learning Composable World Models for ARCcited by 2
  7. 2025Hacia la interpretabilidad de la detecci'on anticipada de riesgos de depresi'on utilizando grandes modelos de lenguajecited by 1
  8. 2025GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoningcited by 1
  9. 2025Vector Symbolic Algebras for the Abstraction and Reasoning Corpuscited by 1
  10. 2024System 2 Reasoning for Human-AI Alignment: Generality and Adaptivity via ARC-AGIcited by 0
  11. 2025Where You Go is Who You Are: Behavioral Theory-Guided LLMs for Inverse Reinforcement Learningcited by 0
  12. 2025Out-of-Context Abduction: LLMs Make Inferences About Procedural Data Leveraging Declarative Facts in Earlier Training Datacited by 0
  13. 2025TCM-5CEval: Extended Deep Evaluation Benchmark for LLM's Comprehensive Clinical Research Competence in Traditional Chinese Medicinecited by 0
  14. 2026JaxARC: A High-Performance JAX-based Environment for Abstraction and Reasoning Researchcited by 0
  15. 2026Exploring Concept Subspace for Self-explainable Text-Attributed Graph Learningcited by 0
  16. 2026Structural Ranking of the Cognitive Plausibility of Computational Models of Analogy and Metaphors with the Minimal Cognitive Gridcited by 0
  17. 2026A Deep Dive into Axiomatic Design -- Part I: Problem Formulationcited by 0
  18. 2026Exactness at Inference: A Representational Criterion for Out-of-Distribution Generalizationcited by 0
  19. 2026Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Modelscited by 0
  20. 2026Anthropomorphism in the age of Large Language Models: An overview of potential risks and mitigationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  2. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  3. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  4. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  5. 2026ReEfBench: Quantifying the Reasoning Efficiency of LLMs
  6. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  7. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond
  8. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning