# Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser

arXiv: https://arxiv.org/abs/2109.02297 · HTML: https://mlsanity.com/paper/2109.02297/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization](https://mlsanity.com/paper/1809.05972/) (2018)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Learning to Select Knowledge for Response Generation in Dialog Systems](https://mlsanity.com/paper/1902.04911/) (2019)
- [Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.06585/) (2017)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [History for Visual Dialog: Do we really need it?](https://mlsanity.com/paper/2005.07493/) (2020)
- [Improving Generative Visual Dialog by Answering Diverse Questions](https://mlsanity.com/paper/1909.10470/) (2019)
- [Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model](https://mlsanity.com/paper/1706.01554/) (2017)
- [End-to-end optimization of goal-driven and visually grounded dialogue systems](https://mlsanity.com/paper/1703.05423/) (2017)
- [Beyond task success: A closer look at jointly learning to see, ask, and GuessWhat](https://mlsanity.com/paper/1809.03408/) (2018)
- [What Should I Ask? Using Conversationally Informative Rewards for Goal-Oriented Visual Dialog](https://mlsanity.com/paper/1907.12021/) (2019)
- [DMRM: A Dual-channel Multi-hop Reasoning Model for Visual Dialog](https://mlsanity.com/paper/1912.08360/) (2019)
- [Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards](https://mlsanity.com/paper/1711.07614/) (2017)
- [Building Task-Oriented Visual Dialog Systems Through Alternative Optimization Between Dialog Policy and Language Generation](https://mlsanity.com/paper/1909.05365/) (2019)
- [Learning Goal-Oriented Visual Dialog via Tempered Policy Gradient](https://mlsanity.com/paper/1807.00737/) (2018)
- [Answer-Driven Visual State Estimator for Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2010.00361/) (2020)

## Built on by (most cited first, 5 of 5)

- [Chatting Makes Perfect: Chat-based Image Retrieval](https://mlsanity.com/paper/2305.20062/) (2023)
- [RT-KGD: Relation Transition Aware Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2207.08212/) (2022)
- [Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning](https://mlsanity.com/paper/2401.04361/) (2024)
- [Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scene](https://mlsanity.com/paper/2203.08362/) (2022)
- [Multi-Modal Dialogue State Tracking for Playing GuessWhich Game](https://mlsanity.com/paper/2408.08431/) (2024)

## Abstract (arXiv)

Considering the importance of building a good Visual Dialog (VD) Questioner, many researchers study the topic under a Q-Bot-A-Bot image-guessing game setting, where the Questioner needs to raise a series of questions to collect information of an undisclosed image. Despite progress has been made in Supervised Learning (SL) and Reinforcement Learning (RL), issues still exist. Firstly, previous methods do not provide explicit and effective guidance for Questioner to generate visually related and informative questions. Secondly, the effect of RL is hampered by an incompetent component, i.e., the Guesser, who makes image predictions based on the generated dialogs and assigns rewards accordingly. To enhance VD Questioner: 1) we propose a Related entity enhanced Questioner (ReeQ) that generates questions under the guidance of related entities and learns entity-based questioning strategy from human dialogs; 2) we propose an Augmented Guesser (AugG) that is strong and is optimized for the VD setting especially. Experimental results on the VisDial v1.0 dataset show that our approach achieves state-of-theart performance on both image-guessing task and question diversity. Human study further proves that our model generates more visually related, informative and coherent questions.