# Learning Visual Knowledge Memory Networks for Visual Question Answering

arXiv: https://arxiv.org/abs/1806.04860 · HTML: https://mlsanity.com/paper/1806.04860/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2018.

## Built on by (most cited first, 10 of 10)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challenges](https://mlsanity.com/paper/2212.13296/) (2022)
- [Endowing Language Models with Multimodal Knowledge Graph Representations](https://mlsanity.com/paper/2206.13163/) (2022)
- [DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation](https://mlsanity.com/paper/2503.13957/) (2025)
- [VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritage](https://mlsanity.com/paper/2510.12750/) (2025)
- [COFAR: Commonsense and Factual Reasoning in Image Search](https://mlsanity.com/paper/2210.08554/) (2022)
- [Improved RAMEN: Towards Domain Generalization for Visual Question Answering](https://mlsanity.com/paper/2109.02370/) (2021)
- [Knowledge-Enhanced Hierarchical Information Correlation Learning for Multi-Modal Rumor Detection](https://mlsanity.com/paper/2306.15946/) (2023)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)

## Abstract (arXiv)

Visual question answering (VQA) requires joint comprehension of images and natural language questions, where many questions can't be directly or clearly answered from visual content but require reasoning from structured human knowledge with confirmation from visual content. This paper proposes visual knowledge memory network (VKMN) to address this issue, which seamlessly incorporates structured human knowledge and deep visual features into memory networks in an end-to-end learning framework. Comparing to existing methods for leveraging external knowledge for supporting VQA, this paper stresses more on two missing mechanisms. First is the mechanism for integrating visual contents with knowledge facts. VKMN handles this issue by embedding knowledge triples (subject, relation, target) and deep visual features jointly into the visual knowledge features. Second is the mechanism for handling multiple knowledge facts expanding from question and answer pairs. VKMN stores joint embedding using key-value pair structure in the memory networks so that it is easy to handle multiple facts. Experiments show that the proposed method achieves promising results on both VQA v1.0 and v2.0 benchmarks, while outperforms state-of-the-art methods on the knowledge-reasoning related questions.