# Dual Semantic Knowledge Composed Multimodal Dialog Systems

arXiv: https://arxiv.org/abs/2305.09990 · HTML: https://mlsanity.com/paper/2305.09990/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 38 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Interest-aware Message-Passing GCN for Recommendation](https://mlsanity.com/paper/2102.10044/) (2021)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [Interacting with Non-Cooperative User: A New Paradigm for Proactive Dialogue Policy](https://mlsanity.com/paper/2204.07433/) (2022)
- [Disentangled Multimodal Representation Learning for Recommendation](https://mlsanity.com/paper/2203.05406/) (2022)
- [Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model](https://mlsanity.com/paper/2207.07934/) (2022)
- [Understanding User Satisfaction with Task-oriented Dialogue Systems](https://mlsanity.com/paper/2204.12195/) (2022)
- [Causal Inference for Knowledge Graph based Recommendation](https://mlsanity.com/paper/2212.10046/) (2022)

## Built on by (most cited first, 4 of 4)

- [Pseudo-triplet Guided Few-shot Composed Image Retrieval](https://mlsanity.com/paper/2407.06001/) (2024)
- [Eval4Sim: An Evaluation Framework for Persona Simulation](https://mlsanity.com/paper/2603.02876/) (2026)
- [Application of frozen large-scale models to multimodal task-oriented dialogue](https://mlsanity.com/paper/2310.00845/) (2023)
- [Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems](https://mlsanity.com/paper/2509.07817/) (2025)

## Abstract (arXiv)

Textual response generation is an essential task for multimodal task-oriented dialog systems.Although existing studies have achieved fruitful progress, they still suffer from two critical limitations: 1) focusing on the attribute knowledge but ignoring the relation knowledge that can reveal the correlations between different entities and hence promote the response generation}, and 2) only conducting the cross-entropy loss based output-level supervision but lacking the representation-level regularization. To address these limitations, we devise a novel multimodal task-oriented dialog system (named MDS-S2). Specifically, MDS-S2 first simultaneously acquires the context related attribute and relation knowledge from the knowledge base, whereby the non-intuitive relation knowledge is extracted by the n-hop graph walk. Thereafter, considering that the attribute knowledge and relation knowledge can benefit the responding to different levels of questions, we design a multi-level knowledge composition module in MDS-S2 to obtain the latent composed response representation. Moreover, we devise a set of latent query variables to distill the semantic information from the composed response representation and the ground truth response representation, respectively, and thus conduct the representation-level semantic regularization. Extensive experiments on a public dataset have verified the superiority of our proposed MDS-S2. We have released the codes and parameters to facilitate the research community.