# DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue

arXiv: https://arxiv.org/abs/2007.03310 · HTML: https://mlsanity.com/paper/2007.03310/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 18 of 18)

- [OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts](https://mlsanity.com/paper/2109.12761/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation](https://mlsanity.com/paper/2109.08478/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [Unified Multimodal Model with Unlikelihood Training for Visual Dialog](https://mlsanity.com/paper/2211.13235/) (2022)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [Iterative Robust Visual Grounding with Masked Reference based Centerpoint Supervision](https://mlsanity.com/paper/2307.12392/) (2023)
- [Modeling Explicit Concerning States for Reinforcement Learning in Visual Dialogue](https://mlsanity.com/paper/2107.05250/) (2021)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)
- [Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations](https://mlsanity.com/paper/2408.06725/) (2024)

## Abstract (arXiv)

Visual Dialogue task requires an agent to be engaged in a conversation with human about an image. The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation. In this paper, we propose a novel generative decoding architecture to generate high-quality responses, which moves away from decoding the whole encoded semantics towards the design that advocates both transparency and flexibility. In this architecture, word generation is decomposed into a series of attention-based information selection steps, performed by the novel recurrent Deliberation, Abandon and Memory (DAM) module. Each DAM module performs an adaptive combination of the response-level semantics captured from the encoder and the word-level semantics specifically selected for generating each word. Therefore, the responses contain more detailed and non-repetitive descriptions while maintaining the semantic accuracy. Furthermore, DAM is flexible to cooperate with existing visual dialogue encoders and adaptive to the encoder structures by constraining the information selection mode in DAM. We apply DAM to three typical encoders and verify the performance on the VisDial v1.0 dataset. Experimental results show that the proposed models achieve new state-of-the-art performance with high-quality responses. The code is available at https://github.com/JXZe/DAM.