mlsanityData to 7 Oct 2026

An Analysis of Visual Question Answering Algorithms

arXiv 1703.09684 ↗First posted 28 Mar 2017cs.CV cs.AI cs.CLcited by 90

Not on the map. Why?

Cited by 30 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In visual question answering (VQA), an algorithm must answer text-based questions about images. While multiple datasets for VQA have been created since late 2014, they all have flaws in both their content and the way algorithms are evaluated on them. As a result, evaluation scores are inflated and predominantly determined by answering easier questions, making it difficult to compare different methods. In this paper, we analyze existing VQA algorithms using a new dataset. It contains over 1.6 million questions organized into 12 different categories. We also introduce questions that are meaningless for a given image to force a VQA system to reason about image content. We propose new evaluation schemes that compensate for over-represented question-types and make it easier to study the strengths and weaknesses of algorithms. We analyze the performance of both baseline and state-of-the-art VQA models, including multi-modal compact bilinear pooling (MCB), neural module networks, and recurrent answering units. Our experiments establish how attention helps certain categories more than others, determine which models work better than others, and explain how simple models (e.g. MLP) can surpass more complex models (MCB) by simply learning to answer large, easy question categories.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 14; 2023: 17; 2024: 21; 2025: 16; 2026: 8.

90 papers.

Sort:

By area:

  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  3. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  4. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  5. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  6. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  7. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  8. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  9. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  10. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 86
  11. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  12. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
38 more
  1. 2022All You May Need for VQA are Image Captionscited by 65
  2. 2023What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?cited by 64
  3. 2022Grounding Answers for Visual Questions Asked by Visually Impaired Peoplecited by 47
  4. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  5. 2024PhD: A ChatGPT-Prompted Visual hallucination Evaluation Datasetcited by 44
  6. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  7. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  8. 2024CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoningcited by 31
  9. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learningcited by 30
  10. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  11. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  12. 2021Recent Advances and Trends in Multimodal Deep Learning: A Reviewcited by 26
  13. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  14. 2021Introspective Distillation for Robust Question Answeringcited by 25
  15. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  16. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  17. 2022Modern Question Answering Datasets and Benchmarks: A Surveycited by 23
  18. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  19. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  20. 2021Multimodal Integration of Human-Like Attention in Visual Question Answeringcited by 20
  21. 2021VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answeringcited by 16
  22. 2024Can MLLMs Understand the Deep Implication Behind Chinese Images?cited by 14
  23. 2025MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decodingcited by 14
  24. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  25. 2024Towards Foundation Models for 3D Vision: How Close Are We?cited by 13
  26. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarkscited by 12
  27. 2022CARETS: A Consistency And Robustness Evaluative Test Suite for VQAcited by 11
  28. 2023An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metricscited by 11
  29. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Modelscited by 10
  30. 2024ING-VP: MLLMs cannot Play Easy Vision-based Games Yetcited by 10
  31. 2022MaXM: Towards Multilingual Visual Question Answeringcited by 9
  32. 2022Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learningcited by 9
  33. 2023Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questionscited by 9
  34. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Typescited by 9
  35. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  36. 2023Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusioncited by 8
  37. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithmscited by 8
  38. 2021General Greedy De-bias Learningcited by 7

50 most cited of 90 All 90 (JSON)