# A Comprehensive Survey of Knowledge-Based Vision Question Answering Systems: The Lifecycle of Knowledge in Visual Reasoning Task

arXiv: https://arxiv.org/abs/2504.17547 · HTML: https://mlsanity.com/paper/2504.17547/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 12 of its 176 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 12 of 12)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [MMKB-RAG: A Multi-Modal Knowledge-Based Retrieval-Augmented Generation Framework](https://mlsanity.com/paper/2504.10074/) (2025)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [Do Dogs have Whiskers? A New Knowledge Base of hasPart Relations](https://mlsanity.com/paper/2006.07510/) (2020)
- [GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2402.02503/) (2024)
- [Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA](https://mlsanity.com/paper/2308.15851/) (2023)
- [Knowledge Condensation and Reasoning for Knowledge-based VQA](https://mlsanity.com/paper/2403.10037/) (2024)
- [Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA](https://mlsanity.com/paper/2406.18839/) (2024)
- [Enabling Collaborative Parametric Knowledge Calibration for Retrieval-Augmented Vision Question Answering](https://mlsanity.com/paper/2504.04065/) (2025)

## Built on by (most cited first, 6 of 6)

- [Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering](https://mlsanity.com/paper/2510.14605/) (2025)
- [CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2602.23952/) (2026)
- [Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding](https://mlsanity.com/paper/2508.16974/) (2025)
- [MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2602.15915/) (2026)
- [Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification](https://mlsanity.com/paper/2606.23881/) (2026)
- [Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting](https://mlsanity.com/paper/2607.00159/) (2026)

## Abstract (arXiv)

Knowledge-based Vision Question Answering (KB-VQA) extends general Vision Question Answering (VQA) by not only requiring the understanding of visual and textual inputs but also extensive range of knowledge, enabling significant advancements across various real-world applications. KB-VQA introduces unique challenges, including the alignment of heterogeneous information from diverse modalities and sources, the retrieval of relevant knowledge from noisy or large-scale repositories, and the execution of complex reasoning to infer answers from the combined context. With the advancement of Large Language Models (LLMs), KB-VQA systems have also undergone a notable transformation, where LLMs serve as powerful knowledge repositories, retrieval-augmented generators and strong reasoners. Despite substantial progress, no comprehensive survey currently exists that systematically organizes and reviews the existing KB-VQA methods. This survey aims to fill this gap by establishing a structured taxonomy of KB-VQA approaches, and categorizing the systems into main stages: knowledge representation, knowledge retrieval, and knowledge reasoning. By exploring various knowledge integration techniques and identifying persistent challenges, this work also outlines promising future research directions, providing a foundation for advancing KB-VQA models and their applications.