# History for Visual Dialog: Do we really need it?

arXiv: https://arxiv.org/abs/2005.07493 · HTML: https://mlsanity.com/paper/2005.07493/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 30 of 32)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling](https://mlsanity.com/paper/2107.03451/) (2021)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog](https://mlsanity.com/paper/2205.00423/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation](https://mlsanity.com/paper/2109.08478/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [Voices in a Crowd: Searching for Clusters of Unique Perspectives](https://mlsanity.com/paper/2407.14259/) (2024)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution](https://mlsanity.com/paper/2205.14693/) (2022)
- [Going for GOAL: A Resource for Grounded Football Commentaries](https://mlsanity.com/paper/2211.04534/) (2022)
- [A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions](https://mlsanity.com/paper/2403.17545/) (2024)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser](https://mlsanity.com/paper/2109.02297/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Unified Multimodal Model with Unlikelihood Training for Visual Dialog](https://mlsanity.com/paper/2211.13235/) (2022)
- [Resolving References in Visually-Grounded Dialogue via Text Generation](https://mlsanity.com/paper/2309.13430/) (2023)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding](https://mlsanity.com/paper/2212.04205/) (2022)
- [mathbbVD-mathbbGR: Boosting mathbbVisual mathbbDialog with Cascaded Spatial-Temporal Multi-Modal mathbbGRaphs](https://mlsanity.com/paper/2310.16590/) (2023)
- ['What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges](https://mlsanity.com/paper/2307.15554/) (2023)
- [Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models](https://mlsanity.com/paper/2409.14247/) (2024)
- [Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Space](https://mlsanity.com/paper/2106.15217/) (2021)
- [Modeling Explicit Concerning States for Reinforcement Learning in Visual Dialogue](https://mlsanity.com/paper/2107.05250/) (2021)
- [Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge](https://mlsanity.com/paper/2204.06970/) (2022)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Collecting Visually-Grounded Dialogue with A Game Of Sorts](https://mlsanity.com/paper/2309.05162/) (2023)
- [Adversarial Robustness of Visual Dialog](https://mlsanity.com/paper/2207.02639/) (2022)

## Abstract (arXiv)

Visual Dialog involves "understanding" the dialog history (what has been discussed previously) and the current question (what is asked), in addition to grounding information in the image, to generate the correct response. In this paper, we show that co-attention models which explicitly encode dialog history outperform models that don't, achieving state-of-the-art performance (72 % NDCG on val set). However, we also expose shortcomings of the crowd-sourcing dataset collection procedure by showing that history is indeed only required for a small amount of the data and that the current evaluation metric encourages generic replies. To that end, we propose a challenging subset (VisDialConv) of the VisDial val set and provide a benchmark of 63% NDCG.