mlsanityData to 13 Oct 2026

Analyzing the Behavior of Visual Question Answering Models

arXiv 1606.07356 ↗First posted 23 Jun 2016cs.CL cs.AI cs.CVcited by 119

Not on the map. Why?

Abstract (arXiv)

Recently, a number of deep-learning based models have been proposed for the task of Visual Question Answering (VQA). The performance of most models is clustered around 60-70%. In this paper we propose systematic methods to analyze the behavior of these models as a first step towards recognizing their strengths and weaknesses, and identifying the most fruitful directions for progress. We analyze two models, one each from two major classes of VQA models -- with-attention and without-attention and show the similarities and differences in the behavior of these models. We also analyze the winning entry of the VQA Challenge 2016. Our behavior analysis reveals that despite recent progress, today's VQA models are "myopic" (tend to fail on sufficiently novel instances), often "jump to conclusions" (converge on a predicted answer after 'listening' to just half the question), and are "stubborn" (do not change their answers across images).

Key facts. Cited by 119 papers in the map (data through 13 Oct 2026). First posted on arXiv 23 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 37; 2022: 26; 2023: 21; 2024: 15; 2025: 9; 2026: 11.

119 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 677
  2. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 446
  3. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  4. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 144
  5. 2021Learning Debiased Representation via Disentangled Feature Augmentationcited by 116
  6. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  7. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 94
  8. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 90
  9. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  10. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  11. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  12. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 57
38 more
  1. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 56
  2. 2024Probing Visual Language Priors in VLMscited by 44
  3. 20213D Question Answeringcited by 41
  4. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  5. 2022Generalizing to the Future: Mitigating Entity Bias in Fake News Detectioncited by 39
  6. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  7. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  8. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34
  9. 2021Towards Robust Classification Model by Counterfactual and Invariant Data Generationcited by 30
  10. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  11. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  12. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 28
  13. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
  14. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  15. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  16. 2021Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysiscited by 22
  17. 2021Counterfactual Samples Synthesizing and Training for Robust Visual Question Answeringcited by 22
  18. 2023Rationalization for Explainable NLP: A Surveycited by 22
  19. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  20. 2021VisQA: X-raying Vision and Language Reasoning in Transformerscited by 19
  21. 2022Revisiting the Importance of Amplifying Bias for Debiasingcited by 19
  22. 2021Check It Again: Progressive Visual Question Answering via Visual Entailmentcited by 18
  23. 2021Ethics Sheets for AI Taskscited by 18
  24. 2021Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Visioncited by 18
  25. 2022Saving Dense Retriever from Shortcut Dependency in Conversational Searchcited by 18
  26. 2022PACS: A Dataset for Physical Audiovisual CommonSense Reasoningcited by 18
  27. 2022Learning to Retrieve Videos by Asking Questionscited by 18
  28. 2022Rethinking Data Augmentation for Robust Visual Question Answeringcited by 18
  29. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  30. 2021GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answeringcited by 16
  31. 2021VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answeringcited by 16
  32. 2021Uncertainty Calibration for Ensemble-Based Debiasing Methodscited by 16
  33. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  34. 2022Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instancescited by 16
  35. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  36. 2021A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanationscited by 14
  37. 2021Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answeringcited by 14
  38. 2021Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Imagescited by 14

50 most cited of 119 All 119 (JSON)