mlsanityData to 7 Oct 2026

Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation

arXiv 1701.08251 ↗First posted 28 Jan 2017cs.CL cs.AI cs.CVcited by 59

Not on the map. Why?

Abstract (arXiv)

The popularity of image sharing on social media and the engagement it creates between users reflects the important role that visual context plays in everyday conversations. We present a novel task, Image-Grounded Conversations (IGC), in which natural-sounding conversations are generated about a shared image. To benchmark progress, we introduce a new multiple-reference dataset of crowd-sourced, event-centric conversations on images. IGC falls on the continuum between chit-chat and goal-directed conversation models, where visual grounding constrains the topic of conversation to event-driven utterances. Experiments with models trained on social media data show that the combination of visual and textual context enhances the quality of generated conversational turns. In human evaluation, the gap between human performance and that of both neural and retrieval architectures suggests that multi-modal IGC presents an interesting challenge for dialogue research.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jan 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 10; 2023: 21; 2024: 8; 2025: 6; 2026: 2.

59 papers.

Sort:

By area:

  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  2. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  3. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  4. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  5. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  6. 2024Reliable Conflictive Multi-View Learningcited by 45
  7. 2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43
  8. 2023See, Say, and Segment: Teaching LMMs to Overcome False Premisescited by 40
  9. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  10. 2021Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledgecited by 30
  11. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  12. 2021Constructing Multi-Modal Dialogue Dataset by Replacing Text with Semantically Relevant Imagescited by 26
38 more
  1. 2021Focused Attention Improves Document-Grounded Generationcited by 24
  2. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  3. 2021Maria: A Visual Experience Powered Conversational Agentcited by 19
  4. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  5. 2022DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Datasetcited by 17
  6. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  7. 2021Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmarkcited by 12
  8. 2024cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Paperscited by 10
  9. 2023BIG-C: a Multimodal Multi-Purpose Dataset for Bembacited by 9
  10. 2021Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generationcited by 8
  11. 2024Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledgecited by 8
  12. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  13. 2025Do What? Teaching Vision-Language-Action Models to Reject the Impossiblecited by 7
  14. 2022Persona-Based Conversational AI: State of the Art and Challengescited by 6
  15. 2021The JDDC 2.0 Corpus: A Large-Scale Multimodal Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Servicecited by 5
  16. 2022Towards Building an Open-Domain Dialogue System Incorporated with Internet Memescited by 5
  17. 2023Large Language Models can Share Images, Too!cited by 5
  18. 2023A Survey of Personality, Persona, and Profile in Conversational Agents and Chatbotscited by 5
  19. 2022CAISE: Conversational Agent for Image Search and Editingcited by 3
  20. 2023ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialoguecited by 3
  21. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
  22. 2021The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialoguescited by 2
  23. 2022Learning to Express in Knowledge-Grounded Conversationcited by 2
  24. 2022Building Knowledge-Grounded Dialogue Systems with Graph-Based Semantic Modelingcited by 2
  25. 2023Models of symbol emergence in communication: a conceptual review and a guide for avoiding local minimacited by 2
  26. 2023Building Multimodal AI Chatbotscited by 2
  27. 2023ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generationcited by 2
  28. 2023Persona-Coded Poly-Encoder: Persona-Guided Multi-Stream Conversational Sentence Scoringcited by 2
  29. 2024SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interactioncited by 2
  30. 2022Lexical Knowledge Internalization for Neural Dialog Generationcited by 1
  31. 2023Which One Are You Referring To? Multimodal Object Identification in Situated Dialoguecited by 1
  32. 2023A Unified Framework for Slot based Response Generation in a Multimodal Dialogue Systemcited by 1
  33. 2023Visual Question Generation in Bengalicited by 1
  34. 2023Context Retrieval via Normalized Contextual Latent Interaction for Conversational Agentcited by 1
  35. 2024An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generationcited by 1
  36. 2024VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Taskscited by 1
  37. 2025KwaiChat: A Large-Scale Video-Driven Multilingual Mixed-Type Dialogue Corpuscited by 1
  38. 2025CoMAC: Conversational Agent for Multi-Source Auxiliary Context with Sparse and Symmetric Latent Interactionscited by 1

50 most cited of 59 All 59 (JSON)