# Explicit Knowledge-based Reasoning for Visual Question Answering

arXiv: https://arxiv.org/abs/1511.02570 · HTML: https://mlsanity.com/paper/1511.02570/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2015.

## Built on by (most cited first, 30 of 85)

- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?](https://mlsanity.com/paper/2302.11713/) (2023)
- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [KAT: A Knowledge Augmented Transformer for Vision-and-Language](https://mlsanity.com/paper/2112.08614/) (2021)
- [REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory](https://mlsanity.com/paper/2212.05221/) (2022)
- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [Multi-Modal Answer Validation for Knowledge-Based VQA](https://mlsanity.com/paper/2103.12248/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning](https://mlsanity.com/paper/2301.05226/) (2023)
- [Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering](https://mlsanity.com/paper/2109.04014/) (2021)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning](https://mlsanity.com/paper/2202.04800/) (2022)
- [Zero-shot Visual Question Answering using Knowledge Graph](https://mlsanity.com/paper/2107.05348/) (2021)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.09138/) (2022)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [DISCOS: Bridging the Gap between Discourse Knowledge and Commonsense Knowledge](https://mlsanity.com/paper/2101.00154/) (2021)
- [Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2109.08029/) (2021)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge](https://mlsanity.com/paper/2101.06013/) (2021)
- [Modern Question Answering Datasets and Benchmarks: A Survey](https://mlsanity.com/paper/2206.15030/) (2022)
- [A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering](https://mlsanity.com/paper/2311.07536/) (2023)
- [HRVQA: A Visual Question Answering Benchmark for High-Resolution Aerial Images](https://mlsanity.com/paper/2301.09460/) (2023)
- [Cross-Modal Retrieval Augmentation for Multi-Modal Classification](https://mlsanity.com/paper/2104.08108/) (2021)

## Abstract (arXiv)

We describe a method for visual question answering which is capable of reasoning about contents of an image on the basis of information extracted from a large-scale knowledge base. The method not only answers natural language questions using concepts not contained in the image, but can provide an explanation of the reasoning by which it developed its answer. The method is capable of answering far more complex questions than the predominant long short-term memory-based approach, and outperforms it significantly in the testing. We also provide a dataset and a protocol by which to evaluate such methods, thus addressing one of the key issues in general visual ques- tion answering.