# Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations

arXiv: https://arxiv.org/abs/2408.06725 · HTML: https://mlsanity.com/paper/2408.06725/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline](https://mlsanity.com/paper/1912.02379/) (2019)
- [VD-BERT: A Unified Vision and Dialog Transformer with BERT](https://mlsanity.com/paper/2004.13278/) (2020)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [History for Visual Dialog: Do we really need it?](https://mlsanity.com/paper/2005.07493/) (2020)
- [DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue](https://mlsanity.com/paper/1911.07251/) (2019)
- [Dual Attention Networks for Visual Reference Resolution in Visual Dialog](https://mlsanity.com/paper/1902.09368/) (2019)
- [Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model](https://mlsanity.com/paper/1706.01554/) (2017)
- [UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog](https://mlsanity.com/paper/2205.00423/) (2022)
- [DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue](https://mlsanity.com/paper/2007.03310/) (2020)
- [Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning](https://mlsanity.com/paper/1711.07613/) (2017)
- [DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialog](https://mlsanity.com/paper/1912.08360/) (2019)
- [Visual Dialogue State Tracking for Question Generation](https://mlsanity.com/paper/1911.07928/) (2019)
- [KBGN: Knowledge-Bridge Graph Network for Adaptive Vision-Text Reasoning in Visual Dialogue](https://mlsanity.com/paper/2008.04858/) (2020)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Guessing State Tracking for Visual Dialogue](https://mlsanity.com/paper/2002.10340/) (2020)
- [Multi-View Attention Network for Visual Dialog](https://mlsanity.com/paper/2004.14025/) (2020)
- [SeqDialN: Sequential Visual Dialog Networks in Joint Visual-Linguistic Representation Space](https://mlsanity.com/paper/2008.00397/) (2020)

## Abstract (arXiv)

Visual Dialog (VD) is a task where an agent answers a series of image-related questions based on a multi-round dialog history. However, previous VD methods often treat the entire dialog history as a simple text input, disregarding the inherent conversational information flows at the round level. In this paper, we introduce Multi-round Dialogue State Tracking model (MDST), a framework that addresses this limitation by leveraging the dialogue state learned from dialog history to answer questions. MDST captures each round of dialog history, constructing internal dialogue state representations defined as 2-tuples of vision-language representations. These representations effectively ground the current question, enabling the generation of accurate answers. Experimental results on the VisDial v1.0 dataset demonstrate that MDST achieves a new state-of-the-art performance in generative setting. Furthermore, through a series of human studies, we validate the effectiveness of MDST in generating long, consistent, and human-like answers while consistently answering a series of questions correctly.