# Do AI Models Perform Human-like Abstract Reasoning Across Modalities?

arXiv: https://arxiv.org/abs/2510.02125 · HTML: https://mlsanity.com/paper/2510.02125/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 16 of 16)

- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [ARC Prize 2024: Technical Report](https://mlsanity.com/paper/2412.04604/) (2024)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [A Benchmark for Compositional Visual Reasoning](https://mlsanity.com/paper/2206.05379/) (2022)
- [H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark](https://mlsanity.com/paper/2409.01374/) (2024)
- [How Far Are We from Intelligent Visual Deductive Reasoning?](https://mlsanity.com/paper/2403.04732/) (2024)
- [Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems](https://mlsanity.com/paper/2411.01173/) (2024)
- [Response: Emergent analogical reasoning in large language models](https://mlsanity.com/paper/2308.16118/) (2023)
- [Neural networks for abstraction and reasoning: Towards broad generalization in machines](https://mlsanity.com/paper/2402.03507/) (2024)
- [Bongard-RWR+: Real-World Representations of Fine-Grained Concepts in Bongard Problems](https://mlsanity.com/paper/2508.12026/) (2025)
- [On Data Synthesis and Post-training for Visual Abstract Reasoning](https://mlsanity.com/paper/2504.01324/) (2025)

## Built on by (most cited first, 10 of 10)

- [AI Coding Agents Can Reproduce Social Science Findings](https://mlsanity.com/paper/2606.11447/) (2026)
- [Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education](https://mlsanity.com/paper/2602.12196/) (2026)
- [Do LLMs Build Spatial World Models? Evidence from Grid-World Maze Tasks](https://mlsanity.com/paper/2604.10690/) (2026)
- [Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs](https://mlsanity.com/paper/2606.28574/) (2026)
- [Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks](https://mlsanity.com/paper/2609.21181/) (2026)
- [Can Consumer Chatbots Reason? A Student-Led Field Experiment Embedded in an "AI-for-All" Undergraduate Course](https://mlsanity.com/paper/2601.04225/) (2025)
- [Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction](https://mlsanity.com/paper/2607.08233/) (2026)
- [PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks](https://mlsanity.com/paper/2609.27288/) (2026)
- [From Understanding to Creation: A Prerequisite-Free AI Literacy Course with Technical Depth Across Majors](https://mlsanity.com/paper/2604.09634/) (2026)
- [LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB](https://mlsanity.com/paper/2604.14437/) (2026)

## Abstract (arXiv)

OpenAI's o3-preview reasoning model exceeded human accuracy on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models recognize and reason with the abstractions the benchmark was designed to test? Here we investigate abstraction abilities of AI models using the closely related but simpler ConceptARC benchmark. Our evaluations vary input modality (textual vs. visual), use of external Python tools, and reasoning effort. Beyond output accuracy, we evaluate the natural-language rules that models generate to explain their solutions, enabling us to assess whether models recognize the abstractions that ConceptARC was designed to elicit. We show that the best models' rules are frequently based on surface-level ``shortcuts,'' capturing intended abstractions considerably less often than humans. In the visual modality, AI models' output accuracy drops sharply; however, our rule-level analysis reveals that a substantial share of their rules capture the intended abstractions, even as the models struggle to apply these concepts to generate correct solutions. In short, we show that using accuracy alone to evaluate abstract reasoning can substantially overestimate AI capabilities in textual modalities and underestimate it in visual modalities. Our results offer a more faithful picture of AI models' abstract reasoning abilities and a more principled way to track progress toward human-like, abstraction-centered intelligence.