mlsanityData to 7 Oct 2026

The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain

arXiv 2305.07141 ↗First posted 11 May 2023cs.LG cs.AIcited by 97

Not on the map. Why?

Cited by 47 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The abilities to form and abstract concepts is key to human intelligence, but such abilities remain lacking in state-of-the-art AI systems. There has been substantial research on conceptual abstraction in AI, particularly using idealized domains such as Raven's Progressive Matrices and Bongard problems, but even when AI systems succeed on such problems, the systems are rarely evaluated in depth to see if they have actually grasped the concepts they are meant to capture. In this paper we describe an in-depth evaluation benchmark for the Abstraction and Reasoning Corpus (ARC), a collection of few-shot abstraction and analogy problems developed by Chollet [2019]. In particular, we describe ConceptARC, a new, publicly available benchmark in the ARC domain that systematically assesses abstraction and generalization abilities on a number of basic spatial and semantic concepts. ConceptARC differs from the original ARC dataset in that it is specifically organized around "concept groups" -- sets of problems that focus on specific concepts and that are vary in complexity and level of abstraction. We report results on testing humans on this benchmark as well as three machine solvers: the top two programs from a 2021 ARC competition and OpenAI's GPT-4. Our results show that humans substantially outperform the machine solvers on this benchmark, showing abilities to abstract and generalize concepts that are not yet captured by AI systems. We believe that this benchmark will spur improvements in the development of AI systems for conceptual abstraction and in the effective evaluation of such systems.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 17 of its 44 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

17 of 44 references.

References · 17

  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2019Unmasking Clever Hans Predictors and Assessing What Machines Really Learncited by 359
  3. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  4. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  5. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  6. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  7. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  8. 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
  9. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  10. 2021Communicating Natural Programs to Humans and Machinescited by 35
  11. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33
  12. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
5 more
  1. 2022Graphs, Constraints, and Search for the Abstraction and Reasoning Corpuscited by 32
  2. 2021Neural-guided, Bidirectional Program Search for Abstraction and Reasoningcited by 23
  3. 2021First Steps of an Approach to the ARC Challenge based on Descriptive Grid Models and the Minimum Description Length Principlecited by 13
  4. 2022Evaluating Understanding on Conceptual Abstraction Benchmarkscited by 13
  5. 2021Using Program Synthesis and Inductive Logic Programming to solve Bongard Problemscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 31; 2025: 32; 2026: 22.

97 papers.

Sort:

By area:

  1. 2023Large Language Models as General Pattern Machinescited by 196
  2. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  3. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 81
  4. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 67
  5. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 54
  6. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patternscited by 54
  7. 2024Combining Induction and Transduction for Abstract Reasoningcited by 53
  8. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  9. 2024Evolving Code with A Large Language Modelcited by 42
  10. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 40
  11. 2025Less is More: Recursive Reasoning with Tiny Networkscited by 38
  12. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
38 more
  1. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  2. 2024H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmarkcited by 32
  3. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  4. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  5. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 29
  6. 2024CodeIt: Self-Improving Language Models with Prioritized Hindsight Replaycited by 23
  7. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Modelscited by 22
  8. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  9. 2025Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspectivecited by 17
  10. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?cited by 16
  11. 2026LoopViT: Scaling Visual ARC with Looped Transformerscited by 14
  12. 2024Limits of Transformer Language Models on Learning to Compose Algorithmscited by 13
  13. 2023Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformercited by 12
  14. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  15. 2024Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neurosciencecited by 11
  16. 2024Generalized Planning for the Abstraction and Reasoning Corpuscited by 10
  17. 2024What is it for a Machine Learning Model to Have a Capability?cited by 10
  18. 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 10
  19. 2025Do AI Models Perform Human-like Abstract Reasoning Across Modalities?cited by 10
  20. 2024Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objectscited by 9
  21. 2024Can Large Language Models generalize analogy solving like children can?cited by 9
  22. 2023Solving ARC visual analogies with neural embeddings and vector arithmetic: A generalized methodcited by 8
  23. 2024ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)cited by 8
  24. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 8
  25. 2024Do Large Language Models Solve ARC Visual Analogies Like People Do?cited by 7
  26. 2024ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learningcited by 7
  27. 2024Creativity in AI: Progresses and Challengescited by 7
  28. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  29. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehuntscited by 7
  30. 2025From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Modelscited by 7
  31. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
  32. 2023ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?cited by 6
  33. 2023In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?cited by 6
  34. 2024Generating Diverse Hypotheses for Inductive Reasoningcited by 6
  35. 2025Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendmentcited by 6
  36. 2025Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferencescited by 6
  37. 2025Think Visually, Reason Textually: Vision-Language Synergy in ARCcited by 6
  38. 2023Building Trust in Conversational AI: A Comprehensive Review and Solution Architecture for Explainable, Privacy-Aware Systems using LLMs and Knowledge Graphcited by 5

50 most cited of 97 All 97 (JSON)