# DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialog

arXiv: https://arxiv.org/abs/1912.08360 · HTML: https://mlsanity.com/paper/1912.08360/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 14 of 14)

- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages](https://mlsanity.com/paper/2305.04160/) (2023)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation](https://mlsanity.com/paper/2109.08478/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser](https://mlsanity.com/paper/2109.02297/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Affective Feedback Synthesis Towards Multimodal Text and Image Data](https://mlsanity.com/paper/2203.12692/) (2022)
- [Grounding is All You Need? Dual Temporal Grounding for Video Dialog](https://mlsanity.com/paper/2410.05767/) (2024)
- [Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scene](https://mlsanity.com/paper/2203.08362/) (2022)
- [Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations](https://mlsanity.com/paper/2408.06725/) (2024)

## Abstract (arXiv)

Visual Dialog is a vision-language task that requires an AI agent to engage in a conversation with humans grounded in an image. It remains a challenging task since it requires the agent to fully understand a given question before making an appropriate response not only from the textual dialog history, but also from the visually-grounded information. While previous models typically leverage single-hop reasoning or single-channel reasoning to deal with this complex multimodal reasoning task, which is intuitively insufficient. In this paper, we thus propose a novel and more powerful Dual-channel Multi-hop Reasoning Model for Visual Dialog, named DMRM. DMRM synchronously captures information from the dialog history and the image to enrich the semantic representation of the question by exploiting dual-channel reasoning. Specifically, DMRM maintains a dual channel to obtain the question- and history-aware image features and the question- and image-aware dialog history features by a mulit-hop reasoning process in each channel. Additionally, we also design an effective multimodal attention to further enhance the decoder to generate more accurate responses. Experimental results on the VisDial v0.9 and v1.0 datasets demonstrate that the proposed model is effective and outperforms compared models by a significant margin.