Intelligence Analysis of Language Models
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
In this project, we test the effectiveness of Large Language Models (LLMs) on the Abstraction and Reasoning Corpus (ARC) dataset. This dataset serves as a representative benchmark for testing abstract reasoning abilities, requiring a fundamental understanding of key concepts such as object identification, basic counting, and elementary geometric principles. Tasks from this dataset are converted into a prompt-based format for evaluation. Initially, we assess the models' potential through a Zero-shot approach. Subsequently, we investigate the application of the Chain-of-Thought (CoT) technique, aiming to determine its role in improving model performance. Our results suggest that, despite the high expectations placed on contemporary LLMs, these models still struggle in non-linguistic domains, even when dealing with simpler subsets of the ARC dataset. Our study is the first to concentrate on the capabilities of open-source models in this context. The code, dataset, and prompts supporting this project's findings can be found in our GitHub repository, accessible at: https://github.com/Lianga2000/LLMsOnARC.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 16 references that are in the map. First posted on arXiv 20 Jul 2024.
Builds on
8 of 16 references.
Closest ancestors · 1
Same direction, newest first
Other references · 7
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
- 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
- 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 81
- 2022Graphs, Constraints, and Search for the Abstraction and Reasoning Corpuscited by 32
- 2021Neural-guided, Bidirectional Program Search for Abstraction and Reasoningcited by 23
- 2020Neural Abstract Reasonercited by 7
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
- 2025Reasoning Capabilities and Invariability of Large Language Models
- 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus
- 2023Large Language Models Are Not Strong Abstract Reasoners
- 2025Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective
- 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
- 2023ThoughtSource: A central hub for large language model reasoning data
- 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models