# A-I-RAVEN and I-RAVEN-Mesh: Two New Benchmarks for Abstract Visual Reasoning

arXiv: https://arxiv.org/abs/2406.11061 · HTML: https://mlsanity.com/paper/2406.11061/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain](https://mlsanity.com/paper/2305.07141/) (2023)
- [Stratified Rule-Aware Network for Abstract Visual Reasoning](https://mlsanity.com/paper/2002.06838/) (2020)
- [Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning](https://mlsanity.com/paper/2010.00763/) (2020)
- [Learning Perceptual Inference by Contrasting](https://mlsanity.com/paper/1912.00086/) (2019)
- [Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks](https://mlsanity.com/paper/2311.09247/) (2023)
- [Learning to Make Analogies by Contrasting Abstract Relational Structure](https://mlsanity.com/paper/1902.00120/) (2019)
- [The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning](https://mlsanity.com/paper/2007.04212/) (2020)
- [Scale-Localized Abstract Reasoning](https://mlsanity.com/paper/2009.09405/) (2020)
- [Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution](https://mlsanity.com/paper/2103.14230/) (2021)
- [Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matrices](https://mlsanity.com/paper/2201.12382/) (2022)
- [Learning Representations that Support Extrapolation](https://mlsanity.com/paper/2007.05059/) (2020)
- [An Explicitly Relational Neural Network Architecture](https://mlsanity.com/paper/1905.10307/) (2019)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning](https://mlsanity.com/paper/2111.12990/) (2021)
- [Multi-Label Contrastive Learning for Abstract Visual Reasoning](https://mlsanity.com/paper/2012.01944/) (2020)
- [Effective Abstract Reasoning with Dual-Contrast Network](https://mlsanity.com/paper/2205.13720/) (2022)
- [Learning to reason over visual objects](https://mlsanity.com/paper/2303.02260/) (2023)
- [Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning](https://mlsanity.com/paper/2004.12193/) (2020)
- [IQ of Neural Networks](https://mlsanity.com/paper/1710.01692/) (2017)
- [Evaluating the Progress of Deep Learning for Visual Relational Concepts](https://mlsanity.com/paper/2001.10857/) (2020)
- [PQA: Perceptual Question Answering](https://mlsanity.com/paper/2104.03589/) (2021)
- [One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems](https://mlsanity.com/paper/2312.09997/) (2023)
- [A Cognitively-Inspired Neural Architecture for Visual Abstract Reasoning Using Contrastive Perceptual and Conceptual Processing](https://mlsanity.com/paper/2309.10532/) (2023)

## Built on by (most cited first, 4 of 4)

- [Toward Cognitive Supersensing in Multimodal Large Language Model](https://mlsanity.com/paper/2602.01541/) (2026)
- [Procedural Refinement by LLM-driven Algorithmic Debugging for ARC-AGI-2](https://mlsanity.com/paper/2603.20334/) (2026)
- [Advancing Generalization Across a Variety of Abstract Visual Reasoning Tasks](https://mlsanity.com/paper/2505.13391/) (2025)
- [Gram-Space: Structure-Preserving Codebook Compression for Memory-Efficient Neuro-Symbolic AI](https://mlsanity.com/paper/2608.01528/) (2026)

## Abstract (arXiv)

We study generalization and knowledge reuse capabilities of deep neural networks in the domain of abstract visual reasoning (AVR), employing Raven's Progressive Matrices (RPMs), a recognized benchmark task for assessing AVR abilities. Two knowledge transfer scenarios referring to the I-RAVEN dataset are investigated. Firstly, inspired by generalization assessment capabilities of the PGM dataset and popularity of I-RAVEN, we introduce Attributeless-I-RAVEN (A-I-RAVEN), a benchmark with 10 generalization regimes that allow to systematically test generalization of abstract rules applied to held-out attributes at various levels of complexity (primary and extended regimes). In contrast to PGM, A-I-RAVEN features compositionality, a variety of figure configurations, and does not require substantial computational resources. Secondly, we construct I-RAVEN-Mesh, a dataset that enriches RPMs with a novel component structure comprising line-based patterns, facilitating assessment of progressive knowledge acquisition in transfer learning setting. We evaluate 13 strong models from the AVR literature on the introduced datasets, revealing their specific shortcomings in generalization and knowledge transfer.