# C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset

arXiv: https://arxiv.org/abs/1704.08243 · HTML: https://mlsanity.com/paper/1704.08243/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2017.

## Built on by (most cited first, 24 of 24)

- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [A Benchmark for Compositional Visual Reasoning](https://mlsanity.com/paper/2206.05379/) (2022)
- [Zero-shot Visual Question Answering using Knowledge Graph](https://mlsanity.com/paper/2107.05348/) (2021)
- [Efficient Data Collection for Robotic Manipulation via Compositional Generalization](https://mlsanity.com/paper/2403.05110/) (2024)
- [COVR: A test-bed for Visually Grounded Compositional Generalization with real images](https://mlsanity.com/paper/2109.10613/) (2021)
- [Neural-Logic Human-Object Interaction Detection](https://mlsanity.com/paper/2311.09817/) (2023)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.12191/) (2022)
- [CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding](https://mlsanity.com/paper/2311.03354/) (2023)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [Directional Gradient Projection for Robust Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2502.15895/) (2025)
- [Semantically Distributed Robust Optimization for Vision-and-Language Inference](https://mlsanity.com/paper/2110.07165/) (2021)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Compositional Causal Reasoning Evaluation in Language Models](https://mlsanity.com/paper/2503.04556/) (2025)
- [VaseVQA: Multimodal Agent and Benchmark for Ancient Greek Pottery](https://mlsanity.com/paper/2509.17191/) (2025)
- [KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence](https://mlsanity.com/paper/2103.00519/) (2021)
- [A Surprising Failure? Multimodal LLMs and the NLVR Challenge](https://mlsanity.com/paper/2402.17793/) (2024)
- [Attribute Diversity Determines the Systematicity Gap in VQA](https://mlsanity.com/paper/2311.08695/) (2023)
- [A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models](https://mlsanity.com/paper/2309.02691/) (2023)
- [Improved RAMEN: Towards Domain Generalization for Visual Question Answering](https://mlsanity.com/paper/2109.02370/) (2021)
- [Understanding in Artificial Intelligence](https://mlsanity.com/paper/2101.06573/) (2021)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [EKTVQA: Generalized use of External Knowledge to empower Scene Text in Text-VQA](https://mlsanity.com/paper/2108.09717/) (2021)
- [Can you even tell left from right? Presenting a new challenge for VQA](https://mlsanity.com/paper/2203.07664/) (2022)

## Abstract (arXiv)

Visual Question Answering (VQA) has received a lot of attention over the past couple of years. A number of deep learning models have been proposed for this task. However, it has been shown that these models are heavily driven by superficial correlations in the training data and lack compositionality -- the ability to answer questions about unseen compositions of seen concepts. This compositionality is desirable and central to intelligence. In this paper, we propose a new setting for Visual Question Answering where the test question-answer pairs are compositionally novel compared to training question-answer pairs. To facilitate developing models under this setting, we present a new compositional split of the VQA v1.0 dataset, which we call Compositional VQA (C-VQA). We analyze the distribution of questions and answers in the C-VQA splits. Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.