# Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task

arXiv: https://arxiv.org/abs/2502.07190 · HTML: https://mlsanity.com/paper/2502.07190/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Llemma: An Open Language Model For Mathematics](https://mlsanity.com/paper/2310.10631/) (2023)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [Evaluating Language Models for Mathematics through Interactions](https://mlsanity.com/paper/2306.01694/) (2023)
- [Linear algebra with transformers](https://mlsanity.com/paper/2112.01898/) (2021)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark](https://mlsanity.com/paper/2409.01374/) (2024)
- [Language Models as Inductive Reasoners](https://mlsanity.com/paper/2212.10923/) (2022)
- [ANPL: Towards Natural Programming with Interactive Decomposition](https://mlsanity.com/paper/2305.18498/) (2023)
- [Large Language Model (LLM) as a System of Multiple Expert Agents: An Approach to solve the Abstraction and Reasoning Corpus (ARC) Challenge](https://mlsanity.com/paper/2310.05146/) (2023)
- [Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Models](https://mlsanity.com/paper/2401.11725/) (2024)
- [An Approach to Solving the Abstraction and Reasoning Corpus (ARC) Challenge](https://mlsanity.com/paper/2306.03553/) (2023)

## Built on by (most cited first, 4 of 4)

- [The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding](https://mlsanity.com/paper/2502.08946/) (2025)
- [Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation](https://mlsanity.com/paper/2506.02648/) (2025)
- [Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Models](https://mlsanity.com/paper/2507.09506/) (2025)
- [Disentangling generalization and memorization in large language models using chess](https://mlsanity.com/paper/2601.16823/) (2026)

## Abstract (arXiv)

While LLMs have exhibited strong performance on various NLP tasks, it is noteworthy that most of these tasks rely on utilizing the vast amount of knowledge encoded in LLMs' parameters, rather than solving new problems without prior knowledge. In cognitive research, the latter ability is referred to as fluid intelligence, which is considered to be critical for assessing human intelligence. Recent research on fluid intelligence assessments has highlighted significant deficiencies in LLMs' abilities. In this paper, we analyze the challenges LLMs face in demonstrating fluid intelligence through controlled experiments, using the most representative ARC task as an example. Our study revealed three major limitations in existing LLMs: limited ability for skill composition, unfamiliarity with abstract input formats, and the intrinsic deficiency of left-to-right decoding. Our data and code can be found in https://wujunjie1998.github.io/araoc-benchmark.github.io/.