# DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset

arXiv: https://arxiv.org/abs/2212.04119 · HTML: https://mlsanity.com/paper/2212.04119/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 62 references that are in the map.
- First posted on arXiv 8 Dec 2022.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [ParlAI: A Dialog Research Software Platform](https://mlsanity.com/paper/1705.06476/) (2017)
- [Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills](https://mlsanity.com/paper/2004.08449/) (2020)
- [Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search](https://mlsanity.com/paper/2109.05433/) (2021)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [Image Chat: Engaging Grounded Conversations](https://mlsanity.com/paper/1811.00945/) (2018)
- [Fine-grained Image Captioning with CLIP Reward](https://mlsanity.com/paper/2205.13115/) (2022)
- [Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation](https://mlsanity.com/paper/1701.08251/) (2017)
- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog](https://mlsanity.com/paper/1903.03166/) (2019)
- [PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling](https://mlsanity.com/paper/2108.01453/) (2021)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [Visual Reference Resolution using Attention Memory for Visual Dialog](https://mlsanity.com/paper/1709.07992/) (2017)
- [OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts](https://mlsanity.com/paper/2012.15015/) (2020)

## Built on by (most cited first, 17 of 17)

- [MPCHAT: Towards Multimodal Persona-Grounded Conversation](https://mlsanity.com/paper/2305.17388/) (2023)
- [MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets](https://mlsanity.com/paper/2403.03194/) (2024)
- [Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge](https://mlsanity.com/paper/2407.03958/) (2024)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions](https://mlsanity.com/paper/2606.09461/) (2026)
- [Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks](https://mlsanity.com/paper/2606.29082/) (2026)
- [Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions](https://mlsanity.com/paper/2506.00421/) (2025)
- [MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization](https://mlsanity.com/paper/2510.01659/) (2025)
- [PicPersona-TOD : A Dataset for Personalizing Utterance Style in Task-Oriented Dialogue with Image Persona](https://mlsanity.com/paper/2504.17390/) (2025)
- [An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation](https://mlsanity.com/paper/2408.08650/) (2024)
- [PanicToCalm: A Proactive Counseling Agent for Panic Attacks](https://mlsanity.com/paper/2510.21143/) (2025)
- [Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models](https://mlsanity.com/paper/2512.03463/) (2025)
- [GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant](https://mlsanity.com/paper/2603.01059/) (2026)
- [Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges](https://mlsanity.com/paper/2608.17605/) (2026)
- [SynCPKL: Harnessing LLMs to Generate Synthetic Data for Commonsense Persona Knowledge Linking](https://mlsanity.com/paper/2407.15281/) (2024)
- [F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Model](https://mlsanity.com/paper/2508.17714/) (2025)
- [Fine-grained Fragment Retrieval in Multi-modal Long-form Dialogues](https://mlsanity.com/paper/2606.04591/) (2026)

## Abstract (arXiv)

As sharing images in an instant message is a crucial factor, there has been active research on learning an image-text multi-modal dialogue models. However, training a well-generalized multi-modal dialogue model remains challenging due to the low quality and limited diversity of images per dialogue in existing multi-modal dialogue datasets. In this paper, we propose an automated pipeline to construct a multi-modal dialogue dataset, ensuring both dialogue quality and image diversity without requiring minimum human effort. In our pipeline, to guarantee the coherence between images and dialogue, we prompt GPT-4 to infer potential image-sharing moments - specifically, the utterance, speaker, rationale, and image description. Furthermore, we leverage CLIP similarity to maintain consistency between aligned multiple images to the utterance. Through this pipeline, we introduce DialogCC, a high-quality and diverse multi-modal dialogue dataset that surpasses existing datasets in terms of quality and diversity in human evaluation. Our comprehensive experiments highlight that when multi-modal dialogue models are trained using our dataset, their generalization performance on unseen dialogue datasets is significantly enhanced. We make our source code and dataset publicly available.