# Reasoning Visual Dialogs with Structural and Partial Observations

arXiv: https://arxiv.org/abs/1904.05548 · HTML: https://mlsanity.com/paper/1904.05548/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 25 of 25)

- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Navigation Instruction Generation with BEV Perception and Large Language Models](https://mlsanity.com/paper/2407.15087/) (2024)
- [Relation-aware Compositional Zero-shot Learning for Attribute-Object Pair Recognition](https://mlsanity.com/paper/2108.04603/) (2021)
- [UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog](https://mlsanity.com/paper/2205.00423/) (2022)
- [OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts](https://mlsanity.com/paper/2109.12761/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Learning Multi-Attention Context Graph for Group-Based Re-Identification](https://mlsanity.com/paper/2104.14236/) (2021)
- [Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues](https://mlsanity.com/paper/2103.00820/) (2021)
- [VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions](https://mlsanity.com/paper/2305.18756/) (2023)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships](https://mlsanity.com/paper/2203.14260/) (2022)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [Ensemble of MRR and NDCG models for Visual Dialog](https://mlsanity.com/paper/2104.07511/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Unified Multimodal Model with Unlikelihood Training for Visual Dialog](https://mlsanity.com/paper/2211.13235/) (2022)
- [InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Models](https://mlsanity.com/paper/2312.13503/) (2023)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [mathbbVD-mathbbGR: Boosting mathbbVisual mathbbDialog with Cascaded Spatial-Temporal Multi-Modal mathbbGRaphs](https://mlsanity.com/paper/2310.16590/) (2023)
- [One for All: One-stage Referring Expression Comprehension with Dynamic Reasoning](https://mlsanity.com/paper/2208.00361/) (2022)
- [Neuro-Symbolic Visual Dialog](https://mlsanity.com/paper/2208.10353/) (2022)
- [Rich Action-semantic Consistent Knowledge for Early Action Prediction](https://mlsanity.com/paper/2201.09169/) (2022)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [IRRGN: An Implicit Relational Reasoning Graph Network for Multi-turn Response Selection](https://mlsanity.com/paper/2212.00482/) (2022)
- [ChartKG: A Knowledge-Graph-Based Representation for Chart Images](https://mlsanity.com/paper/2410.09761/) (2024)
- [Adversarial Robustness of Visual Dialog](https://mlsanity.com/paper/2207.02639/) (2022)

## Abstract (arXiv)

We propose a novel model to address the task of Visual Dialog which exhibits complex dialog structures. To obtain a reasonable answer based on the current question and the dialog history, the underlying semantic dependencies between dialog entities are essential. In this paper, we explicitly formalize this task as inference in a graphical model with partially observed nodes and unknown graph structures (relations in dialog). The given dialog entities are viewed as the observed nodes. The answer to a given question is represented by a node with missing value. We first introduce an Expectation Maximization algorithm to infer both the underlying dialog structures and the missing node values (desired answers). Based on this, we proceed to propose a differentiable graph neural network (GNN) solution that approximates this process. Experiment results on the VisDial and VisDial-Q datasets show that our model outperforms comparative methods. It is also observed that our method can infer the underlying dialog structure for better dialog reasoning.