# Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks

arXiv: https://arxiv.org/abs/1712.00733 · HTML: https://mlsanity.com/paper/1712.00733/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2017.

## Built on by (most cited first, 12 of 12)

- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generation](https://mlsanity.com/paper/2303.11048/) (2023)
- [Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration](https://mlsanity.com/paper/2107.05080/) (2021)
- [Spatial Commonsense Graph for Object Localisation in Partial Scenes](https://mlsanity.com/paper/2203.05380/) (2022)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Semantic-aware Modular Capsule Routing for Visual Question Answering](https://mlsanity.com/paper/2207.10404/) (2022)
- [MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering](https://mlsanity.com/paper/2107.03216/) (2021)
- [Leveraging commonsense for object localisation in partial scenes](https://mlsanity.com/paper/2211.00562/) (2022)
- [Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2312.12723/) (2023)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey](https://mlsanity.com/paper/2412.08158/) (2024)

## Abstract (arXiv)

Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language. Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents. To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks. Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures. Afterwards, we employ dynamic memory networks to attend to the large body of facts in the knowledge graph and images, and then perform reasoning over these facts to generate corresponding answers. Extensive experiments demonstrate that our model not only achieves the state-of-the-art performance in the visual question answering task, but can also answer open-domain questions effectively by leveraging the external knowledge.