# Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

arXiv: https://arxiv.org/abs/1701.08251 · HTML: https://mlsanity.com/paper/1701.08251/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2017.

## Built on by (most cited first, 30 of 59)

- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [Large Language Models are Few-Shot Health Learners](https://mlsanity.com/paper/2305.15525/) (2023)
- [STaR-GATE: Teaching Language Models to Ask Clarifying Questions](https://mlsanity.com/paper/2403.19154/) (2024)
- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [Reliable Conflictive Multi-View Learning](https://mlsanity.com/paper/2402.16897/) (2024)
- [PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling](https://mlsanity.com/paper/2108.01453/) (2021)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge](https://mlsanity.com/paper/2112.08619/) (2021)
- [Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models](https://mlsanity.com/paper/2308.16463/) (2023)
- [Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images](https://mlsanity.com/paper/2107.08685/) (2021)
- [Focused Attention Improves Document-Grounded Generation](https://mlsanity.com/paper/2104.12714/) (2021)
- [MPCHAT: Towards Multimodal Persona-Grounded Conversation](https://mlsanity.com/paper/2305.17388/) (2023)
- [Maria: A Visual Experience Powered Conversational Agent](https://mlsanity.com/paper/2105.13073/) (2021)
- [PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts](https://mlsanity.com/paper/2305.14839/) (2023)
- [DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset](https://mlsanity.com/paper/2212.04119/) (2022)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmark](https://mlsanity.com/paper/2109.01839/) (2021)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)
- [BIG-C: a Multimodal Multi-Purpose Dataset for Bemba](https://mlsanity.com/paper/2305.17202/) (2023)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge](https://mlsanity.com/paper/2407.03958/) (2024)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [Do What? Teaching Vision-Language-Action Models to Reject the Impossible](https://mlsanity.com/paper/2508.16292/) (2025)
- [Persona-Based Conversational AI: State of the Art and Challenges](https://mlsanity.com/paper/2212.03699/) (2022)
- [The JDDC 2.0 Corpus: A Large-Scale Multimodal Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Service](https://mlsanity.com/paper/2109.12913/) (2021)
- [Towards Building an Open-Domain Dialogue System Incorporated with Internet Memes](https://mlsanity.com/paper/2203.03835/) (2022)
- [Large Language Models can Share Images, Too!](https://mlsanity.com/paper/2310.14804/) (2023)
- [A Survey of Personality, Persona, and Profile in Conversational Agents and Chatbots](https://mlsanity.com/paper/2401.00609/) (2023)

## Abstract (arXiv)

The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialogue research.