# A Benchmark for Compositional Visual Reasoning

arXiv: https://arxiv.org/abs/2206.05379 · HTML: https://mlsanity.com/paper/2206.05379/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 11 Jun 2022.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [CLEVRER: CoLlision Events for Video REpresentation and Reasoning](https://mlsanity.com/paper/1910.01442/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017)
- [Analysing Mathematical Reasoning Abilities of Neural Models](https://mlsanity.com/paper/1904.01557/) (2019)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [A Benchmark for Systematic Generalization in Grounded Language Understanding](https://mlsanity.com/paper/2003.05161/) (2020)
- [Learning long-range spatial dependencies with horizontal gated-recurrent units](https://mlsanity.com/paper/1805.08315/) (2018)
- [Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoning](https://mlsanity.com/paper/2010.00763/) (2020)
- [The Scattering Compositional Learner: Discovering Objects, Attributes, Relationships in Analogical Reasoning](https://mlsanity.com/paper/2007.04212/) (2020)
- [Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions](https://mlsanity.com/paper/2205.13803/) (2022)
- [Disentangling neural mechanisms for perceptual grouping](https://mlsanity.com/paper/1906.01558/) (2019)
- [C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset](https://mlsanity.com/paper/1704.08243/) (2017)
- [V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matrices](https://mlsanity.com/paper/1907.12271/) (2019)
- [Tree-structured composition in neural networks without tree-structured architectures](https://mlsanity.com/paper/1506.04834/) (2015)
- [Recurrent neural circuits for contour detection](https://mlsanity.com/paper/2010.15314/) (2020)
- [ReaSCAN: Compositional Reasoning in Language Grounding](https://mlsanity.com/paper/2109.08994/) (2021)
- [Understanding the computational demands underlying visual reasoning](https://mlsanity.com/paper/2108.03603/) (2021)
- [Recurrent Vision Transformer for Solving Visual Reasoning Problems](https://mlsanity.com/paper/2111.14576/) (2021)
- [QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning](https://mlsanity.com/paper/2205.03075/) (2022)

## Built on by (most cited first, 30 of 43)

- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models](https://mlsanity.com/paper/2502.09696/) (2025)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [What is the Visual Cognition Gap between Humans and Multimodal LLMs?](https://mlsanity.com/paper/2406.10424/) (2024)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity](https://mlsanity.com/paper/2503.11557/) (2025)
- [Self-attention in Vision Transformers Performs Perceptual Grouping, Not Attention](https://mlsanity.com/paper/2303.01542/) (2023)
- [Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects](https://mlsanity.com/paper/2406.15955/) (2024)
- [Does Data Scaling Lead to Visual Compositional Generalization?](https://mlsanity.com/paper/2507.07102/) (2025)
- [Strassen Attention, Split VC Dimension and Compositionality in Transformers](https://mlsanity.com/paper/2501.19215/) (2025)
- [Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detection](https://mlsanity.com/paper/2503.03562/) (2025)
- [Do AI Models Perform Human-like Abstract Reasoning Across Modalities?](https://mlsanity.com/paper/2510.02125/) (2025)
- [SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2602.01990/) (2026)
- [Toward Cognitive Supersensing in Multimodal Large Language Model](https://mlsanity.com/paper/2602.01541/) (2026)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [Disentangling and Integrating Relational and Sensory Information in Transformer Architectures](https://mlsanity.com/paper/2405.16727/) (2024)
- [A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs](https://mlsanity.com/paper/2501.13620/) (2025)
- [Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Survey](https://mlsanity.com/paper/2506.04461/) (2025)
- [Take A Step Back: Rethinking the Two Stages in Visual Reasoning](https://mlsanity.com/paper/2407.19666/) (2024)
- [PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding](https://mlsanity.com/paper/2505.20759/) (2025)
- [Zero-shot Compositional Action Recognition with Neural Logic Constraints](https://mlsanity.com/paper/2508.02320/) (2025)
- [The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoning](https://mlsanity.com/paper/2510.04141/) (2025)
- [PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection](https://mlsanity.com/paper/2510.23594/) (2025)
- [Unraveling the geometry of visual relational reasoning](https://mlsanity.com/paper/2502.17382/) (2025)
- [VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs](https://mlsanity.com/paper/2512.21194/) (2025)
- [DiG-bench: Discovery in Games](https://mlsanity.com/paper/2608.12593/) (2026)
- [Benchmarking Robustness of Text-Image Composed Retrieval](https://mlsanity.com/paper/2311.14837/) (2023)
- [GRAFT: GRaPH and Table Reasoning for Textual Alignment -- A Benchmark for Structured Instruction Following and Visual Reasoning](https://mlsanity.com/paper/2508.15690/) (2025)
- [MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning](https://mlsanity.com/paper/2603.12266/) (2026)
- [Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2605.10765/) (2026)

## Abstract (arXiv)

A fundamental component of human vision is our ability to parse complex visual scenes and judge the relations between their constituent objects. AI benchmarks for visual reasoning have driven rapid progress in recent years with state-of-the-art systems now reaching human accuracy on some of these benchmarks. Yet, a major gap remains in terms of the sample efficiency with which humans and AI systems learn new visual reasoning tasks. Humans' remarkable efficiency at learning has been at least partially attributed to their ability to harness compositionality -- such that they can efficiently take advantage of previously gained knowledge when learning new tasks. Here, we introduce a novel visual reasoning benchmark, Compositional Visual Relations (CVR), to drive progress towards the development of more data-efficient learning algorithms. We take inspiration from fluidic intelligence and non-verbal reasoning tests and describe a novel method for creating compositions of abstract rules and associated image datasets at scale. Our proposed benchmark includes measures of sample efficiency, generalization and transfer across task rules, as well as the ability to leverage compositionality. We systematically evaluate modern neural architectures and find that, surprisingly, convolutional architectures surpass transformer-based architectures across all performance measures in most data regimes. However, all computational models are a lot less data efficient compared to humans even after learning informative visual representations using self-supervision. Overall, we hope that our challenge will spur interest in the development of neural architectures that can learn to harness compositionality toward more efficient learning.