# Find The Gap: Knowledge Base Reasoning For Visual Question Answering

arXiv: https://arxiv.org/abs/2404.10226 · HTML: https://mlsanity.com/paper/2404.10226/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 6 of its 42 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 6 of 6)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Rethinking with Retrieval: Faithful Large Language Model Inference](https://mlsanity.com/paper/2301.00303/) (2022)
- [KAT: A Knowledge Augmented Transformer for Vision-and-Language](https://mlsanity.com/paper/2112.08614/) (2021)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering](https://mlsanity.com/paper/2006.09073/) (2020)
- [Explainable High-order Visual Question Reasoning: A New Benchmark and Knowledge-routed Network](https://mlsanity.com/paper/1909.10128/) (2019)

## Built on by (most cited first, 4 of 4)

- [Dynamic Knowledge Integration for Enhanced Vision-Language Reasoning](https://mlsanity.com/paper/2501.08597/) (2025)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [Large Language Models as a Tool for Mining Object Knowledge](https://mlsanity.com/paper/2410.12959/) (2024)
- [A Survey of Task-Oriented Knowledge Graph Reasoning: Status, Applications, and Prospects](https://mlsanity.com/paper/2506.11012/) (2025)

## Abstract (arXiv)

We analyze knowledge-based visual question answering, for which given a question, the models need to ground it into the visual modality and retrieve the relevant knowledge from a given large knowledge base (KB) to be able to answer. Our analysis has two folds, one based on designing neural architectures and training them from scratch, and another based on large pre-trained language models (LLMs). Our research questions are: 1) Can we effectively augment models by explicit supervised retrieval of the relevant KB information to solve the KB-VQA problem? 2) How do task-specific and LLM-based models perform in the integration of visual and external knowledge, and multi-hop reasoning over both sources of information? 3) Is the implicit knowledge of LLMs sufficient for KB-VQA and to what extent it can replace the explicit KB? Our results demonstrate the positive impact of empowering task-specific and LLM models with supervised external and visual knowledge retrieval models. Our findings show that though LLMs are stronger in 1-hop reasoning, they suffer in 2-hop reasoning in comparison with our fine-tuned NN model even if the relevant information from both modalities is available to the model. Moreover, we observed that LLM models outperform the NN model for KB-related questions which confirms the effectiveness of implicit knowledge in LLMs however, they do not alleviate the need for external KB.