# Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering

arXiv: https://arxiv.org/abs/1809.01124 · HTML: https://mlsanity.com/paper/1809.01124/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Sep 2018.

## Built on by (most cited first, 30 of 30)

- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)
- [Multi-Modal Answer Validation for Knowledge-Based VQA](https://mlsanity.com/paper/2103.12248/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [Zero-shot Visual Question Answering using Knowledge Graph](https://mlsanity.com/paper/2107.05348/) (2021)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge](https://mlsanity.com/paper/2101.06013/) (2021)
- [Cross-Modal Retrieval Augmentation for Multi-Modal Classification](https://mlsanity.com/paper/2104.08108/) (2021)
- [A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA](https://mlsanity.com/paper/2206.14989/) (2022)
- [Hypergraph Transformer: Weakly-supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2204.10448/) (2022)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.02985/) (2022)
- [Semantic-aware Modular Capsule Routing for Visual Question Answering](https://mlsanity.com/paper/2207.10404/) (2022)
- [Knowledge Generation for Zero-shot Knowledge-based VQA](https://mlsanity.com/paper/2402.02541/) (2024)
- [Diversify, Rationalize, and Combine: Ensembling Multiple QA Strategies for Zero-shot Knowledge-based VQA](https://mlsanity.com/paper/2406.12746/) (2024)
- [Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant](https://mlsanity.com/paper/2410.19144/) (2024)
- [The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges](https://mlsanity.com/paper/2303.02411/) (2023)
- [Multimodal Prompt Retrieval for Generative Visual Question Answering](https://mlsanity.com/paper/2306.17675/) (2023)
- [A Survey on Interpretable Cross-modal Reasoning](https://mlsanity.com/paper/2309.01955/) (2023)
- [Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2312.12723/) (2023)
- [Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models](https://mlsanity.com/paper/2601.19060/) (2026)
- [Knowledge-Enhanced Hierarchical Information Correlation Learning for Multi-Modal Rumor Detection](https://mlsanity.com/paper/2306.15946/) (2023)
- [IIU: Independent Inference Units for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2408.07989/) (2024)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [EKTVQA: Generalized use of External Knowledge to empower Scene Text in Text-VQA](https://mlsanity.com/paper/2108.09717/) (2021)
- [Unveiling the frontiers of deep learning: innovations shaping diverse domains](https://mlsanity.com/paper/2309.02712/) (2023)

## Abstract (arXiv)

Question answering is an important task for autonomous agents and virtual assistants alike and was shown to support the disabled in efficiently navigating an overwhelming environment. Many existing methods focus on observation-based questions, ignoring our ability to seamlessly combine observed content with general knowledge. To understand interactions with a knowledge base, a dataset has been introduced recently and keyword matching techniques were shown to yield compelling results despite being vulnerable to misconceptions due to synonyms and homographs. To address this issue, we develop a learning-based approach which goes straight to the facts via a learned embedding space. We demonstrate state-of-the-art results on the challenging recently introduced fact-based visual question answering dataset, outperforming competing methods by more than 5%.