# Do Large Language Models Solve ARC Visual Analogies Like People Do?

arXiv: https://arxiv.org/abs/2403.09734 · HTML: https://mlsanity.com/paper/2403.09734/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks](https://mlsanity.com/paper/2311.09247/) (2023)
- [Fast and flexible: Human program induction in abstract reasoning tasks](https://mlsanity.com/paper/2103.05823/) (2021)
- [In-Context Analogical Reasoning with Pre-Trained Language Models](https://mlsanity.com/paper/2305.17626/) (2023)
- [Do large language models solve verbal analogies like children do?](https://mlsanity.com/paper/2310.20384/) (2023)
- [Toward best research practices in AI Psychology](https://mlsanity.com/paper/2312.01276/) (2023)
- [Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learning](https://mlsanity.com/paper/2204.01437/) (2022)

## Built on by (most cited first, 7 of 7)

- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [Combining Induction and Transduction for Abstract Reasoning](https://mlsanity.com/paper/2411.02272/) (2024)
- [ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.00162/) (2025)
- [Can Large Language Models generalize analogy solving like children can?](https://mlsanity.com/paper/2411.02348/) (2024)
- [GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning](https://mlsanity.com/paper/2505.20672/) (2025)
- [Vector Symbolic Algebras for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.08747/) (2025)
- [AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguity](https://mlsanity.com/paper/2509.14171/) (2025)

## Abstract (arXiv)

The Abstraction Reasoning Corpus (ARC) is a visual analogical reasoning test designed for humans and machines (Chollet, 2019). We compared human and large language model (LLM) performance on a new child-friendly set of ARC items. Results show that both children and adults outperform most LLMs on these tasks. Error analysis revealed a similar "fallback" solution strategy in LLMs and young children, where part of the analogy is simply copied. In addition, we found two other error types, one based on seemingly grasping key concepts (e.g., Inside-Outside) and the other based on simple combinations of analogy input matrices. On the whole, "concept" errors were more common in humans, and "matrix" errors were more common in LLMs. This study sheds new light on LLM reasoning ability and the extent to which we can use error analyses and comparisons with human development to understand how LLMs solve visual analogies.