mlsanityData to 7 Oct 2026

The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue

arXiv 1906.01530 ↗First posted 4 Jun 2019cs.CL cs.AI cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

This paper introduces the PhotoBook dataset, a large-scale collection of visually-grounded, task-oriented dialogues in English designed to investigate shared dialogue history accumulating during conversation. Taking inspiration from seminal work on dialogue analysis, we propose a data-collection task formulated as a collaborative game prompting two online participants to refer to images utilising both their visual context as well as previously established referring expressions. We provide a detailed description of the task setup and a thorough analysis of the 2,500 dialogues collected. To further illustrate the novel features of the dataset, we propose a baseline model for reference resolution which uses a simple method to take into account shared information accumulated in a reference chain. Our results show that this information is particularly important to resolve later descriptions and underline the need to develop more sophisticated models of common ground in dialogue interaction.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 7; 2023: 11; 2024: 5; 2025: 6; 2026: 7.

46 papers.

Sort:

By area:

  1. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  2. 2021Grounding 'Grounding' in NLPcited by 44
  3. 2021Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Modelscited by 18
  4. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  5. 2021Reference-Centric Models for Grounded Collaborative Dialoguecited by 15
  6. 2021Draw Me a Flower: Processing and Grounding Abstraction in Natural Languagecited by 14
  7. 2021Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmarkcited by 12
  8. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  9. 2024Grounding Language in Multi-Perspective Referential Communicationcited by 12
  10. 2021Maintaining Common Ground in Dynamic Environmentscited by 9
  11. 2021Analysis of Language Change in Collaborative Instruction Followingcited by 9
  12. 2022MUG: Interactive Multimodal Grounding on User Interfacescited by 9
34 more
  1. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  2. 2024Conversational Grounding: Annotation and Analysis of Grounding Acts and Grounding Unitscited by 8
  3. 2025Measuring How (Not Just Whether) VLMs Build Common Groundcited by 8
  4. 2026LVLMs and Humans Ground Differently in Referential Communicationcited by 8
  5. 2022Towards Pragmatic Production Strategies for Natural Language Generation Taskscited by 7
  6. 2022Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Modelscited by 7
  7. 2023Revisiting Entropy Rate Constancy in Textcited by 7
  8. 2023Speaking the Language of Your Listener: Audience-Aware Adaptation via Plug-and-Play Theory of Mindcited by 7
  9. 2025Post-training for Efficient Communication via Convention Formationcited by 7
  10. 2025Success and Cost Elicit Convention Formation for Efficient Communicationcited by 7
  11. 2023Asking More Informative Questions for Grounded Retrievalcited by 6
  12. 2023Resolving References in Visually-Grounded Dialogue via Text Generationcited by 5
  13. 2023Symbolic Planning and Code Generation for Grounded Dialoguecited by 5
  14. 2022CAISE: Conversational Agent for Image Search and Editingcited by 3
  15. 2023'What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchangescited by 3
  16. 2024COCO is "ALL'' You Need for Visual Instruction Fine-tuningcited by 3
  17. 2021The Interplay of Task Success and Dialogue Quality: An in-depth Evaluation in Task-Oriented Visual Dialoguescited by 2
  18. 2022Modeling Non-Cooperative Dialogue: Theoretical and Empirical Insightscited by 2
  19. 2023A framework for the emergence and analysis of language in social learning agentscited by 2
  20. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 2
  21. 2025I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialoguecited by 2
  22. 2023MindDial: Belief Dynamics Tracking with Theory-of-Mind Modeling for Situated Neural Dialogue Generationcited by 1
  23. 2023Collecting Visually-Grounded Dialogue with A Game Of Sortscited by 1
  24. 2026Seeing Is Not Sharing: Some Vision-Language Models Overestimate Common Ground in Asymmetric Dialoguecited by 1
  25. 2026Investigating Multimodal Informativity under Different Partner Visibility Conditions in Video-Mediated Dialoguecited by 1
  26. 2021We went to look for meaning and all we got were these lousy representations: aspects of meaning representation for computational semanticscited by 0
  27. 2022Spot the Difference: A Cooperative Object-Referring Game in Non-Perfectly Co-Observable Scenecited by 0
  28. 2023Listener Model for the PhotoBook Referential Game with CLIPScores as Implicit Reference Chaincited by 0
  29. 2025Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Modelscited by 0
  30. 2025Ad hoc conventions generalize to new referentscited by 0
  31. 2026Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actionscited by 0
  32. 2026Automatic Analysis of Collaboration Through Human Conversational Data Resources: A Reviewcited by 0
  33. 2026Toward Measuring Structural Drift in LLM Communication Loopscited by 0
  34. 2026The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialoguecited by 0