# Multi-Modal Open-Domain Dialogue

arXiv: https://arxiv.org/abs/2010.01082 · HTML: https://mlsanity.com/paper/2010.01082/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration](https://mlsanity.com/paper/2306.09093/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images](https://mlsanity.com/paper/2107.08685/) (2021)
- [PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts](https://mlsanity.com/paper/2305.14839/) (2023)
- [DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset](https://mlsanity.com/paper/2212.04119/) (2022)
- [CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos](https://mlsanity.com/paper/2303.09713/) (2023)
- [PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model](https://mlsanity.com/paper/2203.17090/) (2022)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [Large Language Models can Share Images, Too!](https://mlsanity.com/paper/2310.14804/) (2023)
- [Post Turing: Mapping the landscape of LLM Evaluation](https://mlsanity.com/paper/2311.02049/) (2023)
- [ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue](https://mlsanity.com/paper/2305.13602/) (2023)
- [i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data](https://mlsanity.com/paper/2305.12311/) (2023)
- [DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever](https://mlsanity.com/paper/2401.01076/) (2024)
- [Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models](https://mlsanity.com/paper/2407.03615/) (2024)
- [An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generation](https://mlsanity.com/paper/2408.08650/) (2024)
- [On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrieval](https://mlsanity.com/paper/2506.11499/) (2025)

## Abstract (arXiv)

Recent work in open-domain conversational agents has demonstrated that significant improvements in model engagingness and humanness metrics can be achieved via massive scaling in both pre-training data and model size (Adiwardana et al., 2020; Roller et al., 2020). However, if we want to build agents with human-like abilities, we must expand beyond handling just text. A particularly important topic is the ability to see images and communicate about what is perceived. With the goal of engaging humans in multi-modal dialogue, we investigate combining components from state-of-the-art open-domain dialogue agents with those from state-of-the-art vision models. We study incorporating different image fusion schemes and domain-adaptive pre-training and fine-tuning strategies, and show that our best resulting model outperforms strong existing models in multi-modal dialogue while simultaneously performing as well as its predecessor (text-only) BlenderBot (Roller et al., 2020) in text-based conversation. We additionally investigate and incorporate safety components in our final model, and show that such efforts do not diminish model performance with respect to engagingness metrics.