mlsanityData to 14 Oct 2026

MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation

arXiv 2211.05719 ↗First posted 10 Nov 2022cs.CL cs.AI cs.CVcited by 56

Not on the map. Why?

Abstract (arXiv)

Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog dataset to better facilitate multi-modal conversation. MMDialog is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 88x. Second, it contains massive topics to generalize the open-domain. To build engaging dialogue system with this dataset, we propose and normalize two response producing tasks based on retrieval and generative scenarios. In addition, we build two baselines for above tasks with state-of-the-art techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset and scripts are available in https://github.com/victorsungo/MMDialog.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 48 of its 70 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

48 of 70 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2016Improved Techniques for Training GANscited by 2,970
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  7. 2016Modeling Context in Referring Expressionscited by 1,063
  8. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  9. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  10. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  11. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  12. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
36 more
  1. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  2. 2016Visual Dialogcited by 377
  3. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  4. 2015Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Modelscited by 207
  5. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  6. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  7. 2018Neural Approaches to Conversational AIcited by 165
  8. 2017A Knowledge-Grounded Neural Conversation Modelcited by 136
  9. 2015Neural Responding Machine for Short-Text Conversationcited by 124
  10. 2018A Dataset for Document Grounded Conversationscited by 104
  11. 2016Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbotscited by 103
  12. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  13. 2015A Neural Network Approach to Context-Sensitive Generation of Conversational Responsescited by 98
  14. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  15. 2018Image Chat: Engaging Grounded Conversationscited by 77
  16. 2019Two Causal Principles for Improving Visual Dialogcited by 75
  17. 2018Modeling Multi-turn Conversation with Deep Utterance Aggregationcited by 65
  18. 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
  19. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46
  20. 2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43
  21. 2019Multi-step Reasoning via Recurrent Dual Attention for Visual Dialogcited by 41
  22. 2020Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selectioncited by 36
  23. 2018Recursive Visual Attention in Visual Dialogcited by 34
  24. 2019DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphscited by 33
  25. 2019Conversing by Reading: Contentful Neural Conversation with On-demand Machine Readingcited by 32
  26. 2021Multimodal Dialogue Response Generationcited by 32
  27. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  28. 2020OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contextscited by 27
  29. 2021Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Imagescited by 26
  30. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  31. 2020Profile Consistency Identification for Open-domain Dialogue Agentscited by 22
  32. 2021Maria: A Visual Experience Powered Conversational Agentcited by 19
  33. 2020Open Domain Dialogue Generation with Latent Imagescited by 17
  34. 2021OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contextscited by 16
  35. 2021REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generationcited by 4
  36. 2021Small Changes Make Big Differences: Improving Multi-turn Response Selection in Dialogue Systems via Fine-Grained Contrastive Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 18; 2025: 15; 2026: 10.

56 papers.

Sort:

By area:

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  3. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  4. 2024WavChat: A Survey of Spoken Dialogue Modelscited by 90
  5. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 57
  6. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 54
  7. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  8. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  9. 2024DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generationcited by 26
  10. 2025RadVLM: A Multitask Conversational Vision-Language Model for Radiologycited by 25
  11. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 24
  12. 2024STICKERCONV: Generating Multimodal Empathetic Responses from Scratchcited by 22
38 more
  1. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  2. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
  3. 2022DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Datasetcited by 17
  4. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokenscited by 15
  5. 2023CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videoscited by 14
  6. 2024CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Modelscited by 11
  7. 2024MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasetscited by 10
  8. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  9. 2024Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledgecited by 8
  10. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8
  11. 2024Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzlescited by 8
  12. 2025LVLMs are Bad at Overhearing Human Referential Communicationcited by 8
  13. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  14. 2023Analyzing Multilingual Competency of LLMs in Multi-Turn Instruction Following: A Case Study of Arabiccited by 7
  15. 2025Multi-Party Conversational Agents: A Surveycited by 7
  16. 2023Large Language Models can Share Images, Too!cited by 5
  17. 2025WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Modelscited by 5
  18. 2026M3Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactionscited by 5
  19. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 4
  20. 2023ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialoguecited by 3
  21. 2024Multimodal Contrastive In-Context Learningcited by 3
  22. 2025MuDoC: An Interactive Multimodal Document-grounded Conversational AI Systemcited by 3
  23. 2025Towards a Multimodal Document-grounded Conversational AI System for Educationcited by 3
  24. 2025ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversationscited by 3
  25. 2025Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactionscited by 3
  26. 2025MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarizationcited by 3
  27. 2023M2Chat: Empowering VLM for Multimodal LLM Interleaved Text-Image Generationcited by 2
  28. 2024DialCLIP: Empowering CLIP as Multi-Modal Dialog Retrievercited by 2
  29. 2026MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audiocited by 2
  30. 2024An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generationcited by 1
  31. 2024'No' Matters: Out-of-Distribution Detection in Multimodality Long Dialoguecited by 1
  32. 2025KwaiChat: A Large-Scale Video-Driven Multilingual Mixed-Type Dialogue Corpuscited by 1
  33. 2025Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharingcited by 1
  34. 2026OmniTrace: A Unified Framework for Generation-Time Attribution in Omni-Modal LLMscited by 1
  35. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  36. 2023VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialoguecited by 0
  37. 2023Teaching Text-to-Image Models to Communicate in Dialogcited by 0
  38. 2025MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agentscited by 0

50 most cited of 56 All 56 (JSON)