# Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems

arXiv: https://arxiv.org/abs/2509.07817 · HTML: https://mlsanity.com/paper/2509.07817/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 38 of its 55 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions](https://mlsanity.com/paper/2306.04140/) (2023)
- [Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Context](https://mlsanity.com/paper/1911.10484/) (2019)
- [Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems](https://mlsanity.com/paper/1804.08217/) (2018)
- [CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generation](https://mlsanity.com/paper/2505.04481/) (2025)
- [SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis](https://mlsanity.com/paper/2005.05635/) (2020)
- [An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2210.16773/) (2022)
- [Contrast and Generation Make BART a Good Dialogue Emotion Recognizer](https://mlsanity.com/paper/2112.11202/) (2021)
- [Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval](https://mlsanity.com/paper/2407.15051/) (2024)
- [Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval](https://mlsanity.com/paper/2404.15875/) (2024)
- [Towards Personalized Answer Generation in E-Commerce via Multi-Perspective Preference Modeling](https://mlsanity.com/paper/2112.13556/) (2021)
- [Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Stories](https://mlsanity.com/paper/2109.03754/) (2021)
- [APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generation](https://mlsanity.com/paper/2407.21048/) (2024)

## Abstract (arXiv)

Textual response generation is pivotal for multimodal mboxtask-oriented dialog systems, which aims to generate proper textual responses based on the multimodal context. While existing efforts have demonstrated remarkable progress, there still exist the following limitations: 1) textitneglect of unstructured review knowledge and 2) textitunderutilization of large language models (LLMs). Inspired by this, we aim to fully utilize dual knowledge (textiti.e.,  structured attribute and unstructured review knowledge) with LLMs to promote textual response generation in multimodal task-oriented dialog systems. However, this task is non-trivial due to two key challenges: 1) textitdynamic knowledge type selection and 2) textitintention-response decoupling. To address these challenges, we propose a novel dual knowledge-enhanced two-stage reasoner by adapting LLMs for multimodal dialog systems (named DK2R). To be specific, DK2R first extracts both structured attribute and unstructured review knowledge from external knowledge base given the dialog context. Thereafter, DK2R uses an LLM to evaluate each knowledge type's utility by analyzing LLM-generated provisional probe responses. Moreover, DK2R separately summarizes the intention-oriented key clues via dedicated reasoning, which are further used as auxiliary signals to enhance LLM-based textual response generation. Extensive experiments conducted on a public dataset verify the superiority of DK2R. We have released the codes and parameters.