# Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning

arXiv: https://arxiv.org/abs/1711.07613 · HTML: https://mlsanity.com/paper/1711.07613/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2017.

## Built on by (most cited first, 14 of 14)

- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts](https://mlsanity.com/paper/2109.12761/) (2021)
- [CADRE: A Cascade Deep Reinforcement Learning Framework for Vision-based Autonomous Urban Driving](https://mlsanity.com/paper/2202.08557/) (2022)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Co-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videos](https://mlsanity.com/paper/2103.12346/) (2021)
- [Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation](https://mlsanity.com/paper/2109.08478/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [MAST: Video Polyp Segmentation with a Mixture-Attention Siamese Transformer](https://mlsanity.com/paper/2401.12439/) (2024)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations](https://mlsanity.com/paper/2408.06725/) (2024)

## Abstract (arXiv)

The Visual Dialogue task requires an agent to engage in a conversation about an image with a human. It represents an extension of the Visual Question Answering task in that the agent needs to answer a question about an image, but it needs to do so in light of the previous dialogue that has taken place. The key challenge in Visual Dialogue is thus maintaining a consistent, and natural dialogue while continuing to answer questions correctly. We present a novel approach that combines Reinforcement Learning and Generative Adversarial Networks (GANs) to generate more human-like responses to questions. The GAN helps overcome the relative paucity of training data, and the tendency of the typical MLE-based approach to generate overly terse answers. Critically, the GAN is tightly integrated into the attention mechanism that generates human-interpretable reasons for each answer. This means that the discriminative model of the GAN has the task of assessing whether a candidate answer is generated by a human or not, given the provided reason. This is significant because it drives the generative model to produce high quality answers that are well supported by the associated reasoning. The method also generates the state-of-the-art results on the primary benchmark.