# Iterative Context-Aware Graph Inference for Visual Dialog

arXiv: https://arxiv.org/abs/2004.02194 · HTML: https://mlsanity.com/paper/2004.02194/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Apr 2020.

## Built on by (most cited first, 16 of 16)

- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation](https://mlsanity.com/paper/2109.08478/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [Revisiting Local Descriptor for Improved Few-Shot Classification](https://mlsanity.com/paper/2103.16009/) (2021)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Face-to-Face Contrastive Learning for Social Intelligence Question-Answering](https://mlsanity.com/paper/2208.01036/) (2022)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [mathbbVD-mathbbGR: Boosting mathbbVisual mathbbDialog with Cascaded Spatial-Temporal Multi-Modal mathbbGRaphs](https://mlsanity.com/paper/2310.16590/) (2023)
- [Modeling Coreference Relations in Visual Dialog](https://mlsanity.com/paper/2203.02986/) (2022)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog](https://mlsanity.com/paper/2310.07259/) (2023)
- [VU-BERT: A Unified framework for Visual Dialog](https://mlsanity.com/paper/2202.10787/) (2022)
- [How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Grounding](https://mlsanity.com/paper/2402.19116/) (2024)
- [MatDecompSDF: High-Fidelity 3D Shape and PBR Material Decomposition from Multi-View Images](https://mlsanity.com/paper/2507.04749/) (2025)

## Abstract (arXiv)

Visual dialog is a challenging task that requires the comprehension of the semantic dependencies among implicit visual and textual contexts. This task can refer to the relation inference in a graphical model with sparse contexts and unknown graph structure (relation descriptor), and how to model the underlying context-aware relation inference is critical. To this end, we propose a novel Context-Aware Graph (CAG) neural network. Each node in the graph corresponds to a joint semantic feature, including both object-based (visual) and history-related (textual) context representations. The graph structure (relations in dialog) is iteratively updated using an adaptive top-K message passing mechanism. Specifically, in every message passing step, each node selects the most K relevant nodes, and only receives messages from them. Then, after the update, we impose graph attention on all the nodes to get the final graph embedding and infer the answer. In CAG, each node has dynamic relations in the graph (different related K neighbor nodes), and only the most relevant nodes are attributive to the context-aware relational graph inference. Experimental results on VisDial v0.9 and v1.0 datasets show that CAG outperforms comparative methods. Visualization results further validate the interpretability of our method.