mlsanityData to 14 Oct 2026

Measuring abstract reasoning in neural networks

arXiv 1807.04225 ↗First posted 11 Jul 2018cs.LG stat.MLcited by 166

Not on the map. Why?

Cited by 54 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Whether neural networks can learn abstract reasoning or whether they merely rely on superficial statistics is a topic of recent debate. Here, we propose a dataset and challenge designed to probe abstract reasoning, inspired by a well-known human IQ test. To succeed at this challenge, models must cope with various generalisation `regimes' in which the training and test data differ in clearly-defined ways. We show that popular models such as ResNets perform poorly, even when the training and test sets differ only minimally, and we present a novel architecture, with a structure designed to encourage reasoning, that does significantly better. When we vary the way in which the test questions and training data differ, we find that our model is notably proficient at certain forms of generalisation, but notably weak at others. We further show that the model's ability to generalise improves markedly if it is trained to predict symbolic explanations for its answers. Altogether, we introduce and explore ways to both measure and induce stronger abstract reasoning in neural networks. Our freely-available dataset should motivate further progress in this direction.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Jul 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 37; 2022: 36; 2023: 31; 2024: 30; 2025: 18; 2026: 14.

166 papers.

Sort:

By area:

  1. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  2. 2022Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphscited by 144
  3. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  4. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 115
  5. 2022Disentangled Representation Learningcited by 112
  6. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  7. 2021Size-Invariant Graph Representations for Graph Classification Extrapolationscited by 84
  8. 2021OOD-GNN: Out-of-Distribution Generalized Graph Neural Networkcited by 78
  9. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  10. 2022A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matricescited by 64
  11. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  12. 2021Visual Representation Learning Does Not Generalize Strongly Within the Same Domaincited by 61
38 more
  1. 2022Theory of Graph Neural Networks: Representation and Learningcited by 61
  2. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  3. 2021Neuro-Symbolic Artificial Intelligence: Current Trendscited by 54
  4. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 53
  5. 2021Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Executioncited by 43
  6. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  7. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  8. 2021Tell me why! Explanations support learning relational and causal structurecited by 36
  9. 2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
  10. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  11. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  12. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  13. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  14. 2024What is the Visual Cognition Gap between Humans and Multimodal LLMs?cited by 30
  15. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  16. 2023Joint Learning of Label and Environment Causal Independence for Graph Out-of-Distribution Generalizationcited by 29
  17. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  18. 2021Learning Algebraic Representation for Systematic Generalization in Abstract Reasoningcited by 27
  19. 2023The Expressive Power of Graph Neural Networks: A Surveycited by 27
  20. 2021Differentiable sampling of molecular geometries with uncertainty-based adversarial attackscited by 25
  21. 2021Dynamic Inference with Neural Interpreterscited by 24
  22. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24
  23. 2022Effective Abstract Reasoning with Dual-Contrast Networkcited by 23
  24. 2022Multimodal Analogical Reasoning over Knowledge Graphscited by 21
  25. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 21
  26. 2023Systematic Visual Reasoning through Object-Centric Relational Abstractioncited by 20
  27. 2022Abstraction for Deep Reinforcement Learningcited by 19
  28. 2023Learning to reason over visual objectscited by 19
  29. 2021Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoningcited by 18
  30. 2023Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real Worldcited by 18
  31. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  32. 2021HALMA: Humanlike Abstraction Learning Meets Affordance in Rapid Problem Solvingcited by 15
  33. 2022RelViT: Concept-guided Vision Transformer for Visual Relational Reasoningcited by 15
  34. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  35. 2021Probabilistic Analogical Mapping with Semantic Relation Networkscited by 14
  36. 2024Neural Concept Bindercited by 14
  37. 2021Disentangled Representations from Non-Disentangled Modelscited by 13
  38. 2022Evaluating Understanding on Conceptual Abstraction Benchmarkscited by 13

50 most cited of 166 All 166 (JSON)