# A Simple Baseline for Knowledge-Based Visual Question Answering

arXiv: https://arxiv.org/abs/2310.13570 · HTML: https://mlsanity.com/paper/2310.13570/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [KAT: A Knowledge Augmented Transformer for Vision-and-Language](https://mlsanity.com/paper/2112.08614/) (2021)
- [KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA](https://mlsanity.com/paper/2012.11014/) (2020)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [Explicit Knowledge-based Reasoning for Visual Question Answering](https://mlsanity.com/paper/1511.02570/) (2015)
- [Multi-Modal Answer Validation for Knowledge-Based VQA](https://mlsanity.com/paper/2103.12248/) (2021)

## Built on by (most cited first, 14 of 14)

- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning](https://mlsanity.com/paper/2404.07078/) (2024)
- [Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models](https://mlsanity.com/paper/2311.17095/) (2023)
- [VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2311.16492/) (2023)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Tasks](https://mlsanity.com/paper/2504.13199/) (2025)
- [Gather and Trace: Rethinking Video TextVQA from an Instance-oriented Perspective](https://mlsanity.com/paper/2508.04197/) (2025)
- [From Hindsight to Foresight: Self-Encouraged Hindsight Distillation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2511.11132/) (2025)
- [Efficient Multimodal Planning Agent for Visual Question-Answering](https://mlsanity.com/paper/2601.20676/) (2026)
- [Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum](https://mlsanity.com/paper/2603.05256/) (2026)
- [REAL: Resolving Knowledge Conflicts in Knowledge-Intensive Visual Question Answering via Reasoning-Pivot Alignment](https://mlsanity.com/paper/2602.14065/) (2026)
- [MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2602.15915/) (2026)
- [A Knowledge Noise Mitigation Framework for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2509.09159/) (2025)
- [Table Comprehension in Building Codes using Vision Language Models and Domain-Specific Fine-Tuning](https://mlsanity.com/paper/2511.18306/) (2025)

## Abstract (arXiv)

This paper is on the problem of Knowledge-Based Visual Question Answering (KB-VQA). Recent works have emphasized the significance of incorporating both explicit (through external databases) and implicit (through LLMs) knowledge to answer questions requiring external knowledge effectively. A common limitation of such approaches is that they consist of relatively complicated pipelines and often heavily rely on accessing GPT-3 API. Our main contribution in this paper is to propose a much simpler and readily reproducible pipeline which, in a nutshell, is based on efficient in-context learning by prompting LLaMA (1 and 2) using question-informative captions as contextual information. Contrary to recent approaches, our method is training-free, does not require access to external databases or APIs, and yet achieves state-of-the-art accuracy on the OK-VQA and A-OK-VQA datasets. Finally, we perform several ablation studies to understand important aspects of our method. Our code is publicly available at https://github.com/alexandrosXe/ASimple-Baseline-For-Knowledge-Based-VQA