mlsanityData to 7 Oct 2026

Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

arXiv 1612.00837 ↗First posted 2 Dec 2016cs.CV cs.AI cs.CLcited by 2,498

Not on the map. Why?

Cited by 1,171 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Problems at the intersection of vision and language are of significant importance both as challenging research questions and for the rich set of applications they enable. However, inherent structure in our world and bias in our language tend to be a simpler signal for learning than visual modalities, resulting in models that ignore visual information, leading to an inflated sense of their capability. We propose to counter these language priors for the task of Visual Question Answering (VQA) and make vision (the V in VQA) matter! Specifically, we balance the popular VQA dataset by collecting complementary images such that every question in our balanced dataset is associated with not just a single image, but rather a pair of similar images that result in two different answers to the question. Our dataset is by construction more balanced than the original VQA dataset and has approximately twice the number of image-question pairs. Our complete balanced dataset is publicly available at www.visualqa.org as part of the 2nd iteration of the Visual Question Answering Dataset and Challenge (VQA v2.0). We further benchmark a number of state-of-art VQA models on our balanced dataset. All models perform significantly worse on our balanced dataset, suggesting that these models have indeed learned to exploit language priors. This finding provides the first concrete empirical evidence for what seems to be a qualitative sense among practitioners. Finally, our data collection protocol for identifying complementary images enables us to develop a novel interpretable model, which in addition to providing an answer to the given (image, question) pair, also provides a counter-example based explanation. Specifically, it identifies an image that is similar to the original image, but it believes has a different answer to the same question. This can help in building trust for machines among their users.

Key facts. Cited by 2,498 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 184; 2022: 241; 2023: 415; 2024: 617; 2025: 565; 2026: 475.

2,498 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
38 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2025Gemma 3 Technical Reportcited by 1,686
  3. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  14. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  15. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  16. 2021Transformers in Vision: A Surveycited by 1,009
  17. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  18. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  19. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  20. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  21. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  22. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  24. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  26. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  27. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  28. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  29. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  30. 2024Yi: Open Foundation Models by 01.AIcited by 706
  31. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  32. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  33. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  34. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  35. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  36. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  37. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  38. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646

50 most cited of 2,498 All 2,498 (JSON)