# Open Domain Dialogue Generation with Latent Images

arXiv: https://arxiv.org/abs/2004.01981 · HTML: https://mlsanity.com/paper/2004.01981/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Apr 2020.

## Built on by (most cited first, 17 of 17)

- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)
- [Maria: A Visual Experience Powered Conversational Agent](https://mlsanity.com/paper/2105.13073/) (2021)
- [PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts](https://mlsanity.com/paper/2305.14839/) (2023)
- [A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues](https://mlsanity.com/paper/2305.04530/) (2023)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Stylized Knowledge-Grounded Dialogue Generation via Disentangled Template Rewriting](https://mlsanity.com/paper/2204.05610/) (2022)
- [Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs](https://mlsanity.com/paper/2311.15759/) (2023)
- [A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation](https://mlsanity.com/paper/2402.13587/) (2024)
- [Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation](https://mlsanity.com/paper/2405.10121/) (2024)
- [ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation](https://mlsanity.com/paper/2308.00400/) (2023)
- [Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models](https://mlsanity.com/paper/2407.03615/) (2024)
- [VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialogue](https://mlsanity.com/paper/2309.07387/) (2023)
- [Teaching Text-to-Image Models to Communicate in Dialog](https://mlsanity.com/paper/2309.15516/) (2023)

## Abstract (arXiv)

We consider grounding open domain dialogues with images. Existing work assumes that both an image and a textual context are available, but image-grounded dialogues by nature are more difficult to obtain than textual dialogues. Thus, we propose learning a response generation model with both image-grounded dialogues and textual dialogues by assuming that the visual scene information at the time of a conversation can be represented by an image, and trying to recover the latent images of the textual dialogues through text-to-image generation techniques. The likelihood of the two types of dialogues is then formulated by a response generator and an image reconstructor that are learned within a conditional variational auto-encoding framework. Empirical studies are conducted in both image-grounded conversation and text-based conversation. In the first scenario, image-grounded dialogues, especially under a low-resource setting, can be effectively augmented by textual dialogues with latent images; while in the second scenario, latent images can enrich the content of responses and at the same time keep them relevant to contexts.