mlsanityData to 7 Oct 2026

Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering

arXiv 1712.00377 ↗First posted 1 Dec 2017cs.CV cs.AI cs.CLcited by 337

Not on the map. Why?

Cited by 70 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A number of studies have found that today's Visual Question Answering (VQA) models are heavily driven by superficial correlations in the training data and lack sufficient image grounding. To encourage development of models geared towards the latter, we propose a new setting for VQA where for every question type, train and test sets have different prior distributions of answers. Specifically, we present new splits of the VQA v1 and VQA v2 datasets, which we call Visual Question Answering under Changing Priors (VQA-CP v1 and VQA-CP v2 respectively). First, we evaluate several existing VQA models under this new setting and show that their performance degrades significantly compared to the original VQA setting. Second, we propose a novel Grounded Visual Question Answering model (GVQA) that contains inductive biases and restrictions in the architecture specifically designed to prevent the model from 'cheating' by primarily relying on priors in the training data. Specifically, GVQA explicitly disentangles the recognition of visual concepts present in the image from the identification of plausible answer space for a given question, enabling the model to more robustly generalize across different distributions of answers. GVQA is built off an existing VQA model -- Stacked Attention Networks (SAN). Our experiments demonstrate that GVQA significantly outperforms SAN on both VQA-CP v1 and VQA-CP v2 datasets. Interestingly, it also outperforms more powerful VQA models such as Multimodal Compact Bilinear Pooling (MCB) in several cases. GVQA offers strengths complementary to SAN when trained and evaluated on the original VQA v1 and VQA v2 datasets. Finally, GVQA is more transparent and interpretable than existing VQA models.

Key facts. Cited by 337 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 68; 2022: 68; 2023: 56; 2024: 32; 2025: 34; 2026: 79.

337 papers.

Sort:

By area:

  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  4. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  5. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  6. 2022Towards Adversarial Attack on Vision-Language Pre-training Modelscited by 142
  7. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  8. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  9. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  10. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  11. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  12. 2021Medical Visual Question Answering: A Surveycited by 93
38 more
  1. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  2. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  3. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  4. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  5. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  6. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  7. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  8. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  9. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  10. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  11. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  12. 2022All You May Need for VQA are Image Captionscited by 65
  13. 2021Fairness via Representation Neutralizationcited by 63
  14. 2022REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answeringcited by 58
  15. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  16. 2021Human-Adversarial Visual Question Answeringcited by 57
  17. 2026MIRAGE: The Illusion of Visual Understandingcited by 57
  18. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  19. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  20. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  21. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  22. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
  23. 2024Probing Visual Language Priors in VLMscited by 44
  24. 20213D Question Answeringcited by 41
  25. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  26. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  27. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  28. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  29. 2023VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Biascited by 37
  30. 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
  31. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learningcited by 35
  32. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  33. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32
  34. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  35. 2022Bias Mimicking: A Simple Sampling Approach for Bias Mitigationcited by 30
  36. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  37. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  38. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28

50 most cited of 337 All 337 (JSON)