# H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

arXiv: https://arxiv.org/abs/2409.01374 · HTML: https://mlsanity.com/paper/2409.01374/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 8 of 8)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks](https://mlsanity.com/paper/2311.09247/) (2023)
- [Communicating Natural Programs to Humans and Machines](https://mlsanity.com/paper/2106.07824/) (2021)
- [Fast and flexible: Human program induction in abstract reasoning tasks](https://mlsanity.com/paper/2103.05823/) (2021)

## Built on by (most cited first, 30 of 33)

- [ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems](https://mlsanity.com/paper/2505.11831/) (2025)
- [RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts](https://mlsanity.com/paper/2411.15114/) (2024)
- [ARC Prize 2024: Technical Report](https://mlsanity.com/paper/2412.04604/) (2024)
- [The Surprising Effectiveness of Test-Time Training for Few-Shot Learning](https://mlsanity.com/paper/2411.07279/) (2024)
- [Combining Induction and Transduction for Abstract Reasoning](https://mlsanity.com/paper/2411.02272/) (2024)
- [Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGI](https://mlsanity.com/paper/2507.14172/) (2025)
- [Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective](https://mlsanity.com/paper/2505.07859/) (2025)
- [LoopViT: Scaling Visual ARC with Looped Transformers](https://mlsanity.com/paper/2602.02156/) (2026)
- [Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models](https://mlsanity.com/paper/2606.07157/) (2026)
- [EasyARC: Evaluating Vision Language Models on True Visual Reasoning](https://mlsanity.com/paper/2506.11595/) (2025)
- [Do AI Models Perform Human-like Abstract Reasoning Across Modalities?](https://mlsanity.com/paper/2510.02125/) (2025)
- [ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.00162/) (2025)
- [Whither symbols in the era of advanced neural networks?](https://mlsanity.com/paper/2508.05776/) (2025)
- [Toward Cognitive Supersensing in Multimodal Large Language Model](https://mlsanity.com/paper/2602.01541/) (2026)
- [Diverse Inference and Verification for Advanced Reasoning](https://mlsanity.com/paper/2502.09955/) (2025)
- [Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation](https://mlsanity.com/paper/2506.02648/) (2025)
- [Think Visually, Reason Textually: Vision-Language Synergy in ARC](https://mlsanity.com/paper/2511.15703/) (2025)
- [The ARC of Progress towards AGI: A Living Survey of Abstraction and Reasoning](https://mlsanity.com/paper/2603.13372/) (2026)
- [Quantifying artificial intelligence through algorithmic generalization](https://mlsanity.com/paper/2411.05943/) (2024)
- [Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Task](https://mlsanity.com/paper/2502.07190/) (2025)
- [CausalARC: Abstract Reasoning with Causal World Models](https://mlsanity.com/paper/2509.03636/) (2025)
- [ARC-AGI Without Pretraining](https://mlsanity.com/paper/2512.06104/) (2025)
- [From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark](https://mlsanity.com/paper/2505.17482/) (2025)
- [GIFARC: Synthetic Dataset for Leveraging Human-Intuitive Analogies to Elevate AI Reasoning](https://mlsanity.com/paper/2505.20672/) (2025)
- [Bongards at the Boundary of Perception and Reasoning: Programs or Language?](https://mlsanity.com/paper/2602.03038/) (2026)
- [ARC-TGI: Human-Validated Task Generators with Reasoning Chain Templates for ARC-AGI](https://mlsanity.com/paper/2603.05099/) (2026)
- [Slots, Transitions, Loops: Learning Composable World Models for ARC](https://mlsanity.com/paper/2606.12316/) (2026)
- [Vector Symbolic Algebras for the Abstraction and Reasoning Corpus](https://mlsanity.com/paper/2511.08747/) (2025)
- [VISTA: A Visual Harness for Reasoning in an Interactive World](https://mlsanity.com/paper/2610.02200/) (2026)
- [Addressing and Visualizing Misalignments in Human Task-Solving Trajectories](https://mlsanity.com/paper/2409.14191/) (2024)

## Abstract (arXiv)

The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test challenging out-of-distribution generalization in humans and machines. Since 2019, limited progress has been observed on the challenge using existing artificial intelligence methods. Comparing human and machine performance is important for the validity of the benchmark. While previous work explored how well humans can solve tasks from the ARC benchmark, they either did so using only a subset of tasks from the original dataset, or from variants of ARC, and therefore only provided a tentative estimate of human performance. In this work, we obtain a more robust estimate of human performance by evaluating 1729 humans on the full set of 400 training and 400 evaluation tasks from the original ARC problem set. We estimate that average human performance lies between 73.3% and 77.2% correct with a reported empirical average of 76.2% on the training set, and between 55.9% and 68.9% correct with a reported empirical average of 64.2% on the public evaluation set. However, we also find that 790 out of the 800 tasks were solvable by at least one person in three attempts, suggesting that the vast majority of the publicly available ARC tasks are in principle solvable by typical crowd-workers recruited over the internet. Notably, while these numbers are slightly lower than earlier estimates, human performance still greatly exceeds current state-of-the-art approaches for solving ARC. To facilitate research on ARC, we publicly release our dataset, called H-ARC (human-ARC), which includes all of the submissions and action traces from human participants.