mlsanityData to 7 Oct 2026

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks

arXiv 2311.09247 ↗First posted 14 Nov 2023cs.AI cs.LGcited by 54

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We explore the abstract reasoning abilities of text-only and multimodal versions of GPT-4, using the ConceptARC benchmark [10], which is designed to evaluate robust understanding and reasoning with core-knowledge concepts. We extend the work of Moskvichev et al. [10] by evaluating GPT-4 on more detailed, one-shot prompting (rather than simple, zero-shot prompts) with text versions of ConceptARC tasks, and by evaluating GPT-4V, the multimodal version of GPT-4, on zero- and one-shot prompts using image versions of the simplest tasks. Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 18 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

10 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations26 May

Other references · 9

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  3. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  4. 2023Large Language Models as General Pattern Machinescited by 196
  5. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  6. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
  7. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  8. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 52
  9. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 29; 2025: 16; 2026: 6.

54 papers.

Sort:

By area:

  1. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  2. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 71
  3. 2023Visual cognition in multimodal large language modelscited by 38
  4. 2024H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmarkcited by 32
  5. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  6. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 29
  7. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  8. 2024The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Modelscited by 17
  9. 2024Neural networks for abstraction and reasoning: Towards broad generalization in machinescited by 13
  10. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language modelscited by 12
  11. 2024Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objectscited by 9
  12. 2024ItD: Large Language Models Can Teach Themselves Induction through Deductioncited by 8
38 more
  1. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  2. 2024ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)cited by 8
  3. 2024Do Large Language Models Solve ARC Visual Analogies Like People Do?cited by 7
  4. 2024Look Further Ahead: Testing the Limits of GPT-4 in Path Planningcited by 7
  5. 2024ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learningcited by 7
  6. 2024Creativity in AI: Progresses and Challengescited by 7
  7. 2024ALTA: Compiler-Based Analysis of Transformerscited by 7
  8. 2024MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Treecited by 7
  9. 2024Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoningcited by 7
  10. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  11. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
  12. 2023Image and Data Mining in Reticular Chemistry Using GPT-4Vcited by 6
  13. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  14. 2025Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?cited by 5
  15. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  16. 2025Position: We Need An Algorithmic Understanding of Generative AIcited by 5
  17. 2024Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Modelscited by 4
  18. 2024Exploring Continual Learning of Compositional Generalization in NLIcited by 4
  19. 2024Program Synthesis using Inductive Logic Programming for the Abstraction and Reasoning Corpuscited by 4
  20. 2024A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoningcited by 4
  21. 2024TurtleBench: A Visual Programming Benchmark in Turtle Geometrycited by 4
  22. 2025I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Modelscited by 4
  23. 2023Inherent limitations of LLMs regarding spatial informationcited by 3
  24. 2024Misrepresented Technological Solutions in Imagined Futures: The Origins and Dangers of AI Hype in the Research Communitycited by 3
  25. 2024Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproductioncited by 2
  26. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setupcited by 2
  27. 2025NSA: Neuro-symbolic ARC Challengecited by 2
  28. 2025What is an "Abstract Reasoner"? Revisiting Experiments and Arguments about Large Language Modelscited by 2
  29. 2025Large Language Models Show Signs of Alignment with Human Neurocognition During Abstract Reasoningcited by 2
  30. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networkscited by 1
  31. 2024A Relational Inductive Bias for Dimensional Abstraction in Neural Networkscited by 1
  32. 2025Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerationscited by 1
  33. 2025Anticipatory Evaluation of Language Modelscited by 1
  34. 2025Vector Symbolic Algebras for the Abstraction and Reasoning Corpuscited by 1
  35. 2025Shuttle Between the Instructions and the Parameters of Large Language Modelscited by 0
  36. 2025AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguitycited by 0
  37. 2026Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solvingcited by 0
  38. 2026PM-Nav: Priori-Map Guided Embodied Navigation in Functional Buildingscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Assessing GPT4-V on Structured Reasoning Tasks
  2. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Models
  3. 2023An Approach to Solving the Abstraction and Reasoning Corpus (ARC) Challenge
  4. 2023Mind meets machine: Unravelling GPT-4's cognitive psychology
  5. 2025MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
  6. 2026Exploring Human Behavior During Abstract Rule Inference and Problem Solving with the Cognitive Abstraction and Reasoning Corpus
  7. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
  8. 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge