# Attribute Diversity Determines the Systematicity Gap in VQA

arXiv: https://arxiv.org/abs/2311.08695 · HTML: https://mlsanity.com/paper/2311.08695/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [Compositional Attention Networks for Machine Reasoning](https://mlsanity.com/paper/1803.03067/) (2018)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Inferring and Executing Programs for Visual Reasoning](https://mlsanity.com/paper/1705.03633/) (2017)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers](https://mlsanity.com/paper/2108.12284/) (2021)
- [Learning by Abstraction: The Neural State Machine](https://mlsanity.com/paper/1907.03950/) (2019)
- [Explainable and Explicit Visual Reasoning over Scene Graphs](https://mlsanity.com/paper/1812.01855/) (2018)
- [On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law](https://mlsanity.com/paper/2005.09241/) (2020)
- [CLOSURE: Assessing Systematic Generalization of CLEVR Models](https://mlsanity.com/paper/1912.05783/) (2019)
- [The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization](https://mlsanity.com/paper/2110.07732/) (2021)
- [SafeText: A Benchmark for Exploring Physical Safety in Language Models](https://mlsanity.com/paper/2210.10045/) (2022)
- [Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalization](https://mlsanity.com/paper/2109.02575/) (2021)
- [COVR: A test-bed for Visually Grounded Compositional Generalization with real images](https://mlsanity.com/paper/2109.10613/) (2021)
- [C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset](https://mlsanity.com/paper/1704.08243/) (2017)
- [Separating Skills and Concepts for Novel Visual Question Answering](https://mlsanity.com/paper/2107.09106/) (2021)
- [Iterated learning for emergent systematicity in VQA](https://mlsanity.com/paper/2105.01119/) (2021)
- [Revisiting the Compositional Generalization Abilities of Neural Sequence Models](https://mlsanity.com/paper/2203.07402/) (2022)

## Built on by (most cited first, 3 of 3)

- [No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance](https://mlsanity.com/paper/2404.04125/) (2024)
- [A Taxonomy of Transcendence](https://mlsanity.com/paper/2508.17669/) (2025)
- [Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering](https://mlsanity.com/paper/2604.16487/) (2026)

## Abstract (arXiv)

Although modern neural networks often generalize to new combinations of familiar concepts, the conditions that enable such compositionality have long been an open question. In this work, we study the systematicity gap in visual question answering: the performance difference between reasoning on previously seen and unseen combinations of object attributes. To test, we introduce a novel diagnostic dataset, CLEVR-HOPE. We find that the systematicity gap is not reduced by increasing the quantity of training data, but is reduced by increasing the diversity of training data. In particular, our experiments suggest that the more distinct attribute type combinations are seen during training, the more systematic we can expect the resulting model to be.