mlsanityData to 14 Oct 2026

Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems

arXiv 2509.07817 ↗First posted 9 Sep 2025cs.CL cs.MMcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Textual response generation is pivotal for multimodal mboxtask-oriented dialog systems, which aims to generate proper textual responses based on the multimodal context. While existing efforts have demonstrated remarkable progress, there still exist the following limitations: 1) textitneglect of unstructured review knowledge and 2) textitunderutilization of large language models (LLMs). Inspired by this, we aim to fully utilize dual knowledge (textiti.e., structured attribute and unstructured review knowledge) with LLMs to promote textual response generation in multimodal task-oriented dialog systems. However, this task is non-trivial due to two key challenges: 1) textitdynamic knowledge type selection and 2) textitintention-response decoupling. To address these challenges, we propose a novel dual knowledge-enhanced two-stage reasoner by adapting LLMs for multimodal dialog systems (named DK2R). To be specific, DK2R first extracts both structured attribute and unstructured review knowledge from external knowledge base given the dialog context. Thereafter, DK2R uses an LLM to evaluate each knowledge type's utility by analyzing LLM-generated provisional probe responses. Moreover, DK2R separately summarizes the intention-oriented key clues via dedicated reasoning, which are further used as auxiliary signals to enhance LLM-based textual response generation. Extensive experiments conducted on a public dataset verify the superiority of DK2R. We have released the codes and parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 38 of its 55 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

38 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  12. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
25 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,480
  2. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  3. 2024Yi: Open Foundation Models by 01.AIcited by 719
  4. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  5. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 245
  6. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  7. 2019Task-Oriented Dialog Systems that Consider Multiple Appropriate Responses under the Same Contextcited by 99
  8. 2018Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systemscited by 70
  9. 2025CAD-Llama: Leveraging Large Language Models for Computer-Aided Design Parametric 3D Model Generationcited by 67
  10. 2020SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysiscited by 54
  11. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
  12. 2021Contrast and Generation Make BART a Good Dialogue Emotion Recognizercited by 35
  13. 2024Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrievalcited by 30
  14. 2024Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrievalcited by 24
  15. 2021Towards Personalized Answer Generation in E-Commerce via Multi-Perspective Preference Modelingcited by 16
  16. 2021Memory and Knowledge Augmented Language Models for Inferring Salience in Long-Form Storiescited by 12
  17. 2024APTNESS: Incorporating Appraisal Theory and Emotion Support Strategies for Empathetic Response Generationcited by 12
  18. 2022Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Modelcited by 11
  19. 2024FlashSloth: Lightning Multimodal Large Language Models via Embedded Visual Compressioncited by 11
  20. 2024FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detectorcited by 10
  21. 2024FELLAS: Enhancing Federated Sequential Recommendation with LLM as External Servicescited by 9
  22. 2018Memory-augmented Dialogue Management for Task-oriented Dialogue Systemscited by 8
  23. 2025Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generationcited by 7
  24. 2024Target-constrained Bidirectional Planning for Generation of Target-oriented Proactive Dialoguecited by 6
  25. 2023Dual Semantic Knowledge Composed Multimodal Dialog Systemscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialogue
  2. 2022Joint Reasoning on Hybrid-knowledge sources for Task-Oriented Dialog
  3. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  4. 2022DialoKG: Knowledge-Structure Aware Task-Oriented Dialogue Generation
  5. 2025Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems
  6. 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation
  7. 2024Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
  8. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models