mlsanityData to 14 Oct 2026

Attribute Diversity Determines the Systematicity Gap in VQA

arXiv 2311.08695 ↗First posted 15 Nov 2023cs.LG cs.CL cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

Although modern neural networks often generalize to new combinations of familiar concepts, the conditions that enable such compositionality have long been an open question. In this work, we study the systematicity gap in visual question answering: the performance difference between reasoning on previously seen and unseen combinations of object attributes. To test, we introduce a novel diagnostic dataset, CLEVR-HOPE. We find that the systematicity gap is not reduced by increasing the quantity of training data, but is reduced by increasing the diversity of training data. In particular, our experiments suggest that the more distinct attribute type combinations are seen during training, the more systematic we can expect the resulting model to be.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 15 Nov 2023.

Builds on

32 of 48 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  9. 2018Datasheets for Datasetscited by 1,386
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  11. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  12. 2015Neural Module Networkscited by 376
20 more
  1. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  2. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  3. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
  4. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  5. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  6. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  7. 2019Learning by Abstraction: The Neural State Machinecited by 113
  8. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  9. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  10. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  11. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  12. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  13. 2021Finding needles in a haystack: Sampling Structurally-diverse Training Sets from Synthetic Data for Compositional Generalizationcited by 27
  14. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  15. 2017C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Datasetcited by 24
  16. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  17. 2021Iterated learning for emergent systematicity in VQAcited by 21
  18. 2022Revisiting the Compositional Generalization Abilities of Neural Sequence Modelscited by 20
  19. 2017Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networkscited by 19
  20. 2022Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionalitycited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  2. 2025A Taxonomy of Transcendencecited by 2
  3. 2026Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steeringcited by 0