# PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks

arXiv: https://arxiv.org/abs/2609.27288 · HTML: https://mlsanity.com/paper/2609.27288/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Combining Induction and Transduction for Abstract Reasoning](https://mlsanity.com/paper/2411.02272/) (2024)
- [Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGI](https://mlsanity.com/paper/2507.14172/) (2025)
- [Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation](https://mlsanity.com/paper/2404.07353/) (2024)
- [ARC Is a Vision Problem!](https://mlsanity.com/paper/2511.14761/) (2025)
- [Potemkin Understanding in Large Language Models](https://mlsanity.com/paper/2506.21521/) (2025)
- [Do AI Models Perform Human-like Abstract Reasoning Across Modalities?](https://mlsanity.com/paper/2510.02125/) (2025)
- [ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.00162/) (2025)
- [Cognitive Dark Matter: Measuring What AI Misses](https://mlsanity.com/paper/2603.03414/) (2026)
- [OmniCode: A Benchmark for Evaluating Software Engineering Agents](https://mlsanity.com/paper/2602.02262/) (2026)
- [Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks](https://mlsanity.com/paper/2609.21181/) (2026)

## Built on by (most cited first, 1 of 1)

- [Pretraining and Test-Time Training Budgets Matter for ARC: A Controlled Study of VARC and TRM](https://mlsanity.com/paper/2610.13660/) (2026)

## Abstract (arXiv)

The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format fails to evaluate the diversity of abilities that genuine abstract skill acquisition should enable. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task's underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin employs programmatic methods to generate new task instances and transform given inputs for a given ARC task, enabling dynamic generative sampling beyond fixed input-output pairs. Across five state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 25-52 percentage-point performance gap between standard ARC evaluation and evaluation on PotARCin, and find that multi-dimensional evaluation reorders models that standard accuracy ranks alike. We further investigate effects of generative sampling, difficulty of corruption types, and questions of self-consistency, showing that models frequently contradict their own formalized rule even where they have stated it correctly. We also introduce P-ARC, a held-out hand-crafted test set, on which models achieve 1-8% accuracy across all five dimensions, underscoring the importance of more holistic evaluations of abstract reasoning capabilities.