mlsanityData to 14 Oct 2026

LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

arXiv 2305.18354 ↗First posted 26 May 2023cs.CL cs.AIcited by 82

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Can a Large Language Model (LLM) solve simple abstract reasoning problems? We explore this broad question through a systematic analysis of GPT on the Abstraction and Reasoning Corpus (ARC), a representative benchmark of abstract reasoning ability from limited examples in which solutions require some "core knowledge" of concepts such as objects, goal states, counting, and basic geometry. GPT-4 solves only 13/50 of the most straightforward ARC tasks when using textual encodings for their two-dimensional input-output grids. Our failure analysis reveals that GPT-4's capacity to identify objects and reason about them is significantly influenced by the sequential nature of the text that represents an object within a text encoding of a task. To test this hypothesis, we design a new benchmark, the 1D-ARC, which consists of one-dimensional (array-like) tasks that are more conducive to GPT-based reasoning, and where it indeed performs better than on the (2D) ARC. To alleviate this issue, we propose an object-based representation that is obtained through an external tool, resulting in nearly doubling the performance on solved ARC tasks and near-perfect scores on the easier 1D-ARC. Although the state-of-the-art GPT-4 is unable to "reason" perfectly within non-language domains such as the 1D-ARC or a simple ARC subset, our study reveals that the use of object-based representations can significantly improve its reasoning ability. Visualizations, GPT logs, and data are available at https://khalil-research.github.io/LLM4ARC.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 36 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

18 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Abstract Visual Reasoning Enabled by Language7 Mar

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2022PAL: Program-aided Language Modelscited by 652
  7. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
  8. 2023Augmented Language Models: a Surveycited by 403
  9. 2022Reasoning with Language Model Prompting: A Surveycited by 252
  10. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  11. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 101
  12. 2021Communicating Natural Programs to Humans and Machinescited by 35
5 more
  1. 2022Graphs, Constraints, and Search for the Abstraction and Reasoning Corpuscited by 32
  2. 2021Neural-guided, Bidirectional Program Search for Abstraction and Reasoningcited by 23
  3. 2021First Steps of an Approach to the ARC Challenge based on Descriptive Grid Models and the Minimum Description Length Principlecited by 13
  4. 2022Evaluating Understanding on Conceptual Abstraction Benchmarkscited by 13
  5. 2020Neural Abstract Reasonercited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 30; 2025: 32; 2026: 14.

82 papers.

Sort:

By area:

  1. 2023Large Language Models as General Pattern Machinescited by 197
  2. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
  3. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  4. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  5. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  6. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
  7. 2024Efficient Causal Graph Discovery Using Large Language Modelscited by 53
  8. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  9. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 37
  10. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 36
  11. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  12. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 30
38 more
  1. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  2. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  3. 2024Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMscited by 29
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  5. 2024CodeIt: Self-Improving Language Models with Prioritized Hindsight Replaycited by 23
  6. 2026Large Language Model Reasoning Failurescited by 21
  7. 2024A Systematic Assessment of OpenAI o1-Preview for Higher Order Thinking in Educationcited by 16
  8. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHFcited by 15
  9. 2024HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesiscited by 15
  10. 2025LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoningcited by 12
  11. 2023Large Language Model (LLM) as a System of Multiple Expert Agents: An Approach to solve the Abstraction and Reasoning Corpus (ARC) Challengecited by 11
  12. 2023Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solvingcited by 11
  13. 2024Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answeringcited by 11
  14. 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 11
  15. 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
  16. 2024Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objectscited by 10
  17. 2025Climplicit: Climatic Implicit Embeddings for Global Ecological Taskscited by 10
  18. 2025ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpuscited by 10
  19. 2025Neural Cellular Automata for ARC-AGIcited by 10
  20. 2025PRL: Prompts from Reinforcement Learningcited by 9
  21. 2024C2P: Featuring Large Language Models with Causal Reasoningcited by 8
  22. 2024CAX: Cellular Automata Accelerated in JAXcited by 8
  23. 2024ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)cited by 8
  24. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  25. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 8
  26. 2024Do Large Language Models Solve ARC Visual Analogies Like People Do?cited by 7
  27. 2024ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learningcited by 7
  28. 2024Creativity in AI: Progresses and Challengescited by 7
  29. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
  30. 2024Relational decomposition for program synthesiscited by 6
  31. 2025Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluationcited by 6
  32. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5
  33. 2024Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Modelscited by 4
  34. 2024Learning big logical rules by joining small rulescited by 4
  35. 2024Program Synthesis using Inductive Logic Programming for the Abstraction and Reasoning Corpuscited by 4
  36. 2025Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Taskcited by 4
  37. 2025Reasoning Can Hurt the Inductive Abilities of Large Language Modelscited by 4
  38. 2025Program Synthesis via Test-Time Transductioncited by 4

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective
  2. 2023An Approach to Solving the Abstraction and Reasoning Corpus (ARC) Challenge
  3. 2024Abstraction-of-Thought Makes Language Models Better Reasoners
  4. 2023Large Language Models Are Not Strong Abstract Reasoners
  5. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  6. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
  7. 2026Compositional Neuro-Symbolic Reasoning
  8. 2025Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective