mlsanityData to 7 Oct 2026

COVR: A test-bed for Visually Grounded Compositional Generalization with real images

arXiv 2109.10613 ↗First posted 22 Sep 2021cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

While interest in models that generalize at test time to new compositions has risen in recent years, benchmarks in the visually-grounded domain have thus far been restricted to synthetic images. In this work, we propose COVR, a new test-bed for visually-grounded compositional generalization with real images. To create COVR, we use real images annotated with scene graphs, and propose an almost fully automatic procedure for generating question-answer pairs along with a set of context images. COVR focuses on questions that require complex reasoning, including higher-order operations such as quantification and aggregation. Due to the automatic generation process, COVR facilitates the creation of compositional splits, where models at test time need to generalize to new concepts and compositions in a zero- or few-shot setting. We construct compositional splits using COVR and demonstrate a myriad of cases where state-of-the-art pre-trained language-and-vision models struggle to compositionally generalize.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

28 of 36 references.

References · 28

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
16 more
  1. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  2. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  3. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  4. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  5. 2018Improving Text-to-SQL Evaluation Methodologycited by 133
  6. 2019Learning by Abstraction: The Neural State Machinecited by 113
  7. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  8. 2017Building machines that adapt and compute like brainscited by 99
  9. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  10. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  11. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  12. 2020Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehensioncited by 50
  13. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  14. 2017C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Datasetcited by 24
  15. 2018Visual Referring Expression Recognition: What Do Systems Actually Learn?cited by 20
  16. 2021How Transferable are Reasoning Patterns in VQA?cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 10; 2024: 3; 2025: 1; 2026: 2.

25 papers.

Sort:

By area:

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  2. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  4. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  5. 2023Modular Visual Question Answering via Code Generationcited by 47
  6. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  7. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  8. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  9. 2022Image Retrieval from Contextual Descriptionscited by 26
  10. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Modelscited by 23
  11. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  12. 2023Generating Enhanced Negatives for Training Language-Based Object Detectorscited by 11
13 more
  1. 2022Structurally Diverse Sampling for Sample-Efficient Training and Comprehensive Evaluationcited by 8
  2. 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?cited by 8
  3. 2026MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Gamescited by 7
  4. 2022Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systemscited by 6
  5. 2023Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Modelscited by 6
  6. 2023Recursive Visual Programmingcited by 6
  7. 2023The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Modelscited by 5
  8. 2023Attribute Diversity Determines the Systematicity Gap in VQAcited by 3
  9. 2023Dynamic MOdularized Reasoning for Compositional Structured Explanation Generationcited by 1
  10. 2024Sequential Compositional Generalization in Multimodal Modelscited by 1
  11. 2024VDebugger: Harnessing Execution Feedback for Debugging Visual Programscited by 1
  12. 2026HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilitiescited by 1
  13. 2025Impact of Pretraining Word Co-occurrence on Compositional Generalization in Multimodal Modelscited by 0