mlsanityData to 7 Oct 2026

MMChat: Multi-Modal Chat Dataset on Social Media

arXiv 2108.07154 ↗First posted 16 Aug 2021cs.CL cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Incorporating multi-modal contexts in conversation is important for developing more engaging dialogue systems. In this work, we explore this direction by introducing MMChat: a large-scale Chinese multi-modal dialogue corpus (32.4M raw dialogues and 120.84K filtered dialogues). Unlike previous corpora that are crowd-sourced or collected from fictitious movies, MMChat contains image-grounded dialogues collected from real conversations on social media, in which the sparsity issue is observed. Specifically, image-initiated dialogues in common communications may deviate to some non-image-grounded topics as the conversation proceeds. To better investigate this issue, we manually annotate 100K dialogues from MMChat and further filter the corpus accordingly, which yields MMChat-hf. We develop a benchmark model to address the sparsity issue in dialogue generation tasks by adapting the attention routing mechanism on image features. Experiments demonstrate the usefulness of incorporating image features and the effectiveness of handling the sparsity of image features.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 16 Aug 2021.

Builds on

29 of 38 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  9. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  10. 2016Visual Dialogcited by 377
  11. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  12. 2019Audio-Visual Scene-Aware Dialogcited by 122
17 more
  1. 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 114
  2. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  3. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  4. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  5. 2018Image Chat: Engaging Grounded Conversationscited by 76
  6. 2019A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Datacited by 74
  7. 2019Personalized Dialogue Generation with Diversified Traitscited by 73
  8. 2018Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systemscited by 70
  9. 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
  10. 2019Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methodscited by 55
  11. 2019The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agentscited by 45
  12. 2020Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learningcited by 44
  13. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  14. 2020OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contextscited by 27
  15. 2020Multi-Modal Open-Domain Dialoguecited by 19
  16. 2018Game-Based Video-Context Dialoguecited by 15
  17. 2019All-in-One Image-Grounded Conversational Agentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 10; 2024: 6; 2025: 4; 2026: 2.

30 papers.

Sort:

By area:

  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  2. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  3. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  4. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  5. 2022Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systemscited by 28
  6. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  7. 2022Rethinking and Refining the Distinct Metriccited by 27
  8. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  9. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  10. 2022DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Datasetcited by 17
  11. 2022Estimating Soft Labels for Out-of-Domain Intent Detectioncited by 11
  12. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
18 more
  1. 2022Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialoguecited by 8
  2. 2024Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledgecited by 8
  3. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8
  4. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  5. 2023Large Language Models can Share Images, Too!cited by 5
  6. 2023IMAD: IMage-Augmented multi-modal Dialoguecited by 4
  7. 2026H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactionscited by 3
  8. 2022Semi-Supervised Lifelong Language Learningcited by 1
  9. 2023Dialogue-to-Video Retrievalcited by 1
  10. 2024An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generationcited by 1
  11. 2025KwaiChat: A Large-Scale Video-Driven Multilingual Mixed-Type Dialogue Corpuscited by 1
  12. 2022Accuracy on In-Domain Samples Matters When Building Out-of-Domain detectors: A Reply to Marek et al. (2021)cited by 0
  13. 2023VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialoguecited by 0
  14. 2023EXMODD: An EXplanatory Multimodal Open-Domain Dialogue datasetcited by 0
  15. 2024WarCov -- Large multilabel and multimodal dataset from social platformcited by 0
  16. 2025MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agentscited by 0
  17. 2025F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Modelcited by 0
  18. 2026Fine-grained Fragment Retrieval in Multi-modal Long-form Dialoguescited by 0