# From Shallow to Deep: Compositional Reasoning over Graphs for Visual Question Answering

arXiv: https://arxiv.org/abs/2206.12533 · HTML: https://mlsanity.com/paper/2206.12533/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 25 Jun 2022.

## Builds on (most cited first, 18 of 18)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [DenseCap: Fully Convolutional Localization Networks for Dense Captioning](https://mlsanity.com/paper/1511.07571/) (2015)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Explainable and Explicit Visual Reasoning over Scene Graphs](https://mlsanity.com/paper/1812.01855/) (2018)
- [BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection](https://mlsanity.com/paper/1902.00038/) (2019)
- [Graph-Structured Referring Expression Reasoning in The Wild](https://mlsanity.com/paper/2004.08814/) (2020)
- [Explainable Neural Computation via Stack Neural Module Networks](https://mlsanity.com/paper/1807.08556/) (2018)
- [Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering](https://mlsanity.com/paper/2006.09073/) (2020)
- [Meta Module Network for Compositional Visual Reasoning](https://mlsanity.com/paper/1910.03230/) (2019)
- [Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2009.00145/) (2020)
- [Visual Question Answering with Memory-Augmented Networks](https://mlsanity.com/paper/1707.04968/) (2017)
- [Compact Trilinear Interaction for Visual Question Answering](https://mlsanity.com/paper/1909.11874/) (2019)

## Built on by (most cited first, 3 of 3)

- [A Context-Enhanced Framework for Sequential Graph Reasoning](https://mlsanity.com/paper/2412.09056/) (2024)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [Semantic-aware Dynamic Retrospective-Prospective Reasoning for Event-level Video Question Answering](https://mlsanity.com/paper/2305.08059/) (2023)

## Abstract (arXiv)

In order to achieve a general visual question answering (VQA) system, it is essential to learn to answer deeper questions that require compositional reasoning on the image and external knowledge. Meanwhile, the reasoning process should be explicit and explainable to understand the working mechanism of the model. It is effortless for human but challenging for machines. In this paper, we propose a Hierarchical Graph Neural Module Network (HGNMN) that reasons over multi-layer graphs with neural modules to address the above issues. Specifically, we first encode the image by multi-layer graphs from the visual, semantic and commonsense views since the clues that support the answer may exist in different modalities. Our model consists of several well-designed neural modules that perform specific functions over graphs, which can be used to conduct multi-step reasoning within and between different graphs. Compared to existing modular networks, we extend visual reasoning from one graph to more graphs. We can explicitly trace the reasoning process according to module weights and graph attentions. Experiments show that our model not only achieves state-of-the-art performance on the CRIC dataset but also obtains explicit and explainable reasoning procedures.