# Visual Question Answering: A Survey of Methods and Datasets

arXiv: https://arxiv.org/abs/1607.05910 · HTML: https://mlsanity.com/paper/1607.05910/

## Key facts

- Cited by 123 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jul 2016.

## Built on by (most cited first, 30 of 123)

- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [MapQA: A Dataset for Question Answering on Choropleth Maps](https://mlsanity.com/paper/2211.08545/) (2022)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Grounding Answers for Visual Questions Asked by Visually Impaired People](https://mlsanity.com/paper/2202.01993/) (2022)
- [Biomedical Question Answering: A Survey of Approaches and Challenges](https://mlsanity.com/paper/2102.05281/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment](https://mlsanity.com/paper/2411.18688/) (2024)
- [Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video](https://mlsanity.com/paper/2203.06667/) (2022)
- [COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences](https://mlsanity.com/paper/2106.00969/) (2021)
- [From Easy to Hard: Learning Language-guided Curriculum for Visual Question Answering on Remote Sensing Data](https://mlsanity.com/paper/2205.03147/) (2022)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training](https://mlsanity.com/paper/2503.06486/) (2025)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning](https://mlsanity.com/paper/2503.19900/) (2025)
- [(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering](https://mlsanity.com/paper/2202.09277/) (2022)
- [Change Detection Meets Visual Question Answering](https://mlsanity.com/paper/2112.06343/) (2021)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [AR2-D2:Training a Robot Without a Robot](https://mlsanity.com/paper/2306.13818/) (2023)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering](https://mlsanity.com/paper/2209.10326/) (2022)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge](https://mlsanity.com/paper/2101.06013/) (2021)
- [Multimodality of AI for Education: Towards Artificial General Intelligence](https://mlsanity.com/paper/2312.06037/) (2023)
- [Watching the News: Towards VideoQA Models that can Read](https://mlsanity.com/paper/2211.05588/) (2022)
- [Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imagination](https://mlsanity.com/paper/2210.12261/) (2022)
- [Long-Form Answers to Visual Questions from Blind and Low Vision People](https://mlsanity.com/paper/2408.06303/) (2024)

## Abstract (arXiv)

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.