# DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue

arXiv: https://arxiv.org/abs/1911.07251 · HTML: https://mlsanity.com/paper/1911.07251/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2019.

## Built on by (most cited first, 30 of 30)

- [SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning](https://mlsanity.com/paper/2112.08587/) (2021)
- [Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion](https://mlsanity.com/paper/2408.07303/) (2024)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Cross-border Commodity Pricing Strategy Optimization via Mixed Neural Network for Time Series Analysis](https://mlsanity.com/paper/2408.12115/) (2024)
- [Deep Analysis of Time Series Data for Smart Grid Startup Strategies: A Transformer-LSTM-PSO Model Approach](https://mlsanity.com/paper/2408.12129/) (2024)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networks](https://mlsanity.com/paper/2408.12124/) (2024)
- [VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolution](https://mlsanity.com/paper/2205.14693/) (2022)
- [Ensemble of MRR and NDCG models for Visual Dialog](https://mlsanity.com/paper/2104.07511/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Unified Multimodal Model with Unlikelihood Training for Visual Dialog](https://mlsanity.com/paper/2211.13235/) (2022)
- [Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog](https://mlsanity.com/paper/2204.04680/) (2022)
- [mathbbVD-mathbbGR: Boosting mathbbVisual mathbbDialog with Cascaded Spatial-Temporal Multi-Modal mathbbGRaphs](https://mlsanity.com/paper/2310.16590/) (2023)
- [Intelligent logistics management robot path planning algorithm integrating transformer and GCN network](https://mlsanity.com/paper/2501.02749/) (2025)
- [Modeling Coreference Relations in Visual Dialog](https://mlsanity.com/paper/2203.02986/) (2022)
- [CPLOYO: A Pulmonary Nodule Detection Model with Multi-Scale Feature Fusion and Nonlinear Feature Learning](https://mlsanity.com/paper/2503.10045/) (2025)
- [Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review](https://mlsanity.com/paper/2207.00782/) (2022)
- [Affective Feedback Synthesis Towards Multimodal Text and Image Data](https://mlsanity.com/paper/2203.12692/) (2022)
- [SINGA-Easy: An Easy-to-Use Framework for MultiModal Analysis](https://mlsanity.com/paper/2108.02572/) (2021)
- [VisualTextRank: Unsupervised Graph-based Content Extraction for Automating Ad Text to Image Search](https://mlsanity.com/paper/2108.02725/) (2021)
- [Text is NOT Enough: Integrating Visual Impressions into Open-domain Dialogue Generation](https://mlsanity.com/paper/2109.05778/) (2021)
- [Discovering Personalized Semantics for Soft Attributes in Recommender Systems using Concept Activation Vectors](https://mlsanity.com/paper/2202.02830/) (2022)
- [VU-BERT: A Unified framework for Visual Dialog](https://mlsanity.com/paper/2202.10787/) (2022)
- [Reinforced MOOCs Concept Recommendation in Heterogeneous Information Networks](https://mlsanity.com/paper/2203.11011/) (2022)
- [Explanation-Guided Fairness Testing through Genetic Algorithm](https://mlsanity.com/paper/2205.08335/) (2022)
- [Using Graph Neural Networks for Program Termination](https://mlsanity.com/paper/2207.14648/) (2022)
- [Tribe or Not? Critical Inspection of Group Differences Using TribalGram](https://mlsanity.com/paper/2303.09664/) (2023)
- [Synthesizing Sentiment-Controlled Feedback For Multimodal Text and Image Data](https://mlsanity.com/paper/2402.07640/) (2024)
- [Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations](https://mlsanity.com/paper/2408.06725/) (2024)

## Abstract (arXiv)

Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue involves multiple questions which cover a broad range of visual content that could be related to any objects, relationships or semantics. The key challenge in Visual Dialogue task is thus to learn a more comprehensive and semantic-rich image representation which may have adaptive attentions on the image for variant questions. In this research, we propose a novel model to depict an image from both visual and semantic perspectives. Specifically, the visual view helps capture the appearance-level information, including objects and their relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Futhermore, on top of such multi-view image features, we propose a feature selection framework which is able to adaptively capture question-relevant information hierarchically in fine-grained level. The proposed method achieved state-of-the-art results on benchmark Visual Dialogue datasets. More importantly, we can tell which modality (visual or semantic) has more contribution in answering the current question by visualizing the gate values. It gives us insights in understanding of human cognition in Visual Dialogue.