# VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution

arXiv: https://arxiv.org/abs/2205.14693 · HTML: https://mlsanity.com/paper/2205.14693/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 44 references that are in the map.
- First posted on arXiv 29 May 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training](https://mlsanity.com/paper/1908.06066/) (2019)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [End-to-end Neural Coreference Resolution](https://mlsanity.com/paper/1707.07045/) (2017)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Two Causal Principles for Improving Visual Dialog](https://mlsanity.com/paper/1911.10496/) (2019)
- [Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline](https://mlsanity.com/paper/1912.02379/) (2019)
- [VD-BERT: A Unified Vision and Dialog Transformer with BERT](https://mlsanity.com/paper/2004.13278/) (2020)
- [Visual Coreference Resolution in Visual Dialog using Neural Module Networks](https://mlsanity.com/paper/1809.01816/) (2018)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [GECOR: An End-to-End Generative Ellipsis and Co-reference Resolution Model for Task-Oriented Dialogue](https://mlsanity.com/paper/1909.12086/) (2019)
- [History for Visual Dialog: Do we really need it?](https://mlsanity.com/paper/2005.07493/) (2020)
- [DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue](https://mlsanity.com/paper/1911.07251/) (2019)
- [Dual Attention Networks for Visual Reference Resolution in Visual Dialog](https://mlsanity.com/paper/1902.09368/) (2019)
- [Improving Generative Visual Dialog by Answering Diverse Questions](https://mlsanity.com/paper/1909.10470/) (2019)
- [What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues](https://mlsanity.com/paper/1909.00421/) (2019)
- [Knowledge-aware Pronoun Coreference Resolution](https://mlsanity.com/paper/1907.03663/) (2019)
- [Incorporating Context and External Knowledge for Pronoun Coreference Resolution](https://mlsanity.com/paper/1905.10238/) (2019)
- [Modality-Balanced Models for Visual Dialogue](https://mlsanity.com/paper/2001.06354/) (2020)
- [Probabilistic framework for solving Visual Dialog](https://mlsanity.com/paper/1909.04800/) (2019)

## Built on by (most cited first, 7 of 7)

- [McQueen: a Benchmark for Multimodal Conversational Query Rewrite](https://mlsanity.com/paper/2210.12775/) (2022)
- [Unified Multimodal Model with Unlikelihood Training for Visual Dialog](https://mlsanity.com/paper/2211.13235/) (2022)
- [mathbbVD-mathbbGR: Boosting mathbbVisual mathbbDialog with Cascaded Spatial-Temporal Multi-Modal mathbbGRaphs](https://mlsanity.com/paper/2310.16590/) (2023)
- [MDC-R: The Minecraft Dialogue Corpus with Reference](https://mlsanity.com/paper/2506.22062/) (2025)
- [Take That for Me: Multimodal Exophora Resolution with Interactive Questioning for Ambiguous Out-of-View Instructions](https://mlsanity.com/paper/2508.16143/) (2025)
- [V2Dial: Unification of Video and Visual Dialog via Multimodal Experts](https://mlsanity.com/paper/2503.02063/) (2025)
- [Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures](https://mlsanity.com/paper/2505.11726/) (2025)

## Abstract (arXiv)

The visual dialog task requires an AI agent to interact with humans in multi-round dialogs based on a visual environment. As a common linguistic phenomenon, pronouns are often used in dialogs to improve the communication efficiency. As a result, resolving pronouns (i.e., grounding pronouns to the noun phrases they refer to) is an essential step towards understanding dialogs. In this paper, we propose VD-PCR, a novel framework to improve Visual Dialog understanding with Pronoun Coreference Resolution in both implicit and explicit ways. First, to implicitly help models understand pronouns, we design novel methods to perform the joint training of the pronoun coreference resolution and visual dialog tasks. Second, after observing that the coreference relationship of pronouns and their referents indicates the relevance between dialog rounds, we propose to explicitly prune the irrelevant history rounds in visual dialog models' input. With pruned input, the models can focus on relevant dialog history and ignore the distraction in the irrelevant one. With the proposed implicit and explicit methods, VD-PCR achieves state-of-the-art experimental results on the VisDial dataset.