# Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model

arXiv: https://arxiv.org/abs/2207.07934 · HTML: https://mlsanity.com/paper/2207.07934/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 66 references that are in the map.
- First posted on arXiv 16 Jul 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Gated-Attention Architectures for Task-Oriented Language Grounding](https://mlsanity.com/paper/1706.07230/) (2017)
- [Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/1804.08217/) (2018)
- [End-to-End Task-Completion Neural Dialogue Systems](https://mlsanity.com/paper/1703.01008/) (2017)
- [EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching](https://mlsanity.com/paper/2111.08919/) (2021)
- [Contrast and Generation Make BART a Good Dialogue Emotion Recognizer](https://mlsanity.com/paper/2112.11202/) (2021)
- [AnyFace: Free-style Text-to-Face Synthesis and Manipulation](https://mlsanity.com/paper/2203.15334/) (2022)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [Towards Personalized Answer Generation in E-Commerce via Multi-Perspective Preference Modeling](https://mlsanity.com/paper/2112.13556/) (2021)
- [Conversations with Search Engines: SERP-based Conversational Response Generation](https://mlsanity.com/paper/2004.14162/) (2020)
- [M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers](https://mlsanity.com/paper/2104.01122/) (2021)
- ["What can I cook with these ingredients?" -- Understanding cooking-related information needs in conversational search](https://mlsanity.com/paper/2112.04788/) (2021)
- [Symmetric Regularization based BERT for Pair-wise Semantic Reasoning](https://mlsanity.com/paper/1909.03405/) (2019)
- [HeteroQA: Learning towards Question-and-Answering through Multiple Information Sources via Heterogeneous Graph Modeling](https://mlsanity.com/paper/2112.13597/) (2021)
- [Understanding WeChat User Preferences and "Wow" Diffusion](https://mlsanity.com/paper/2103.02930/) (2021)

## Built on by (most cited first, 11 of 11)

- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation](https://mlsanity.com/paper/2412.05818/) (2024)
- [Recent Developments in Recommender Systems: A Survey](https://mlsanity.com/paper/2306.12680/) (2023)
- [Dual Semantic Knowledge Composed Multimodal Dialog Systems](https://mlsanity.com/paper/2305.09990/) (2023)
- [DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking](https://mlsanity.com/paper/2404.04818/) (2024)
- [Application of frozen large-scale models to multimodal task-oriented dialogue](https://mlsanity.com/paper/2310.00845/) (2023)
- [Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialog](https://mlsanity.com/paper/2310.07259/) (2023)
- [Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models](https://mlsanity.com/paper/2407.03615/) (2024)
- [LA-RCS: LLM-Agent-Based Robot Control System](https://mlsanity.com/paper/2505.18214/) (2025)
- [Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems](https://mlsanity.com/paper/2509.07817/) (2025)
- [OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning](https://mlsanity.com/paper/2603.02658/) (2026)

## Abstract (arXiv)

Text response generation for multimodal task-oriented dialog systems, which aims to generate the proper text response given the multimodal context, is an essential yet challenging task. Although existing efforts have achieved compelling success, they still suffer from two pivotal limitations: 1) overlook the benefit of generative pre-training, and 2) ignore the textual context related knowledge. To address these limitations, we propose a novel dual knowledge-enhanced generative pretrained language model for multimodal task-oriented dialog systems (DKMD), consisting of three key components: dual knowledge selection, dual knowledge-enhanced context learning, and knowledge-enhanced response generation. To be specific, the dual knowledge selection component aims to select the related knowledge according to both textual and visual modalities of the given context. Thereafter, the dual knowledge-enhanced context learning component targets seamlessly integrating the selected knowledge into the multimodal context learning from both global and local perspectives, where the cross-modal semantic relation is also explored. Moreover, the knowledge-enhanced response generation component comprises a revised BART decoder, where an additional dot-product knowledge-decoder attention sub-layer is introduced for explicitly utilizing the knowledge to advance the text response generation. Extensive experiments on a public dataset verify the superiority of the proposed DKMD over state-of-the-art competitors.