# OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts

arXiv: https://arxiv.org/abs/2109.12761 · HTML: https://mlsanity.com/paper/2109.12761/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 75 references that are in the map.
- First posted on arXiv 27 Sep 2021.

## Builds on (most cited first, 30 of 56)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Deep Reinforcement Learning for Dialogue Generation](https://mlsanity.com/paper/1606.01541/) (2016)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems](https://mlsanity.com/paper/1506.08909/) (2015)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [Neural Approaches to Conversational AI](https://mlsanity.com/paper/1809.08267/) (2018)
- [Challenges in Building Intelligent Open-domain Dialog Systems](https://mlsanity.com/paper/1905.05709/) (2019)
- [Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions](https://mlsanity.com/paper/2005.06676/) (2020)
- [A Knowledge-Grounded Neural Conversation Model](https://mlsanity.com/paper/1702.01932/) (2017)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Knowledge-Grounded Dialogue Generation with Pre-trained Language Models](https://mlsanity.com/paper/2010.08824/) (2020)
- [Adversarial Learning for Neural Dialogue Generation](https://mlsanity.com/paper/1701.06547/) (2017)
- [A Neural Network Approach to Context-Sensitive Generation of Conversational Responses](https://mlsanity.com/paper/1506.06714/) (2015)
- [Data Noising as Smoothing in Neural Network Language Models](https://mlsanity.com/paper/1703.02573/) (2017)
- [Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.06585/) (2017)

## Built on by (most cited first, 16 of 16)

- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models](https://mlsanity.com/paper/2308.16463/) (2023)
- [DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset](https://mlsanity.com/paper/2212.04119/) (2022)
- [CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos](https://mlsanity.com/paper/2303.09713/) (2023)
- [VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitions](https://mlsanity.com/paper/2305.18756/) (2023)
- [Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding](https://mlsanity.com/paper/2412.17295/) (2024)
- [Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge](https://mlsanity.com/paper/2407.03958/) (2024)
- [BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2408.05926/) (2024)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [A Survey on Multi-modal Machine Translation: Tasks, Methods and Challenges](https://mlsanity.com/paper/2405.12669/) (2024)
- [Large Language Models can Share Images, Too!](https://mlsanity.com/paper/2310.14804/) (2023)
- [Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactions](https://mlsanity.com/paper/2506.00421/) (2025)
- [KwaiChat: A Large-Scale Video-Driven Multilingual Mixed-Type Dialogue Corpus](https://mlsanity.com/paper/2503.06899/) (2025)
- [EXMODD: An EXplanatory Multimodal Open-Domain Dialogue dataset](https://mlsanity.com/paper/2310.10967/) (2023)
- [MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agents](https://mlsanity.com/paper/2502.05887/) (2025)
- [EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue](https://mlsanity.com/paper/2608.20905/) (2026)

## Abstract (arXiv)

In order to better simulate the real human conversation process, models need to generate dialogue utterances based on not only preceding textual contexts but also visual contexts. However, with the development of multi-modal dialogue learning, the dataset scale gradually becomes a bottleneck. In this report, we release OpenViDial 2.0, a larger-scale open-domain multi-modal dialogue dataset compared to the previous version OpenViDial 1.0. OpenViDial 2.0 contains a total number of 5.6 million dialogue turns extracted from either movies or TV series from different resources, and each dialogue turn is paired with its corresponding visual context. We hope this large-scale dataset can help facilitate future researches on open-domain multi-modal dialog generation, e.g., multi-modal pretraining for dialogue generation.