mlsanityData to 14 Oct 2026

PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling

arXiv 2108.01453 ↗First posted 6 Jul 2021cs.IR cs.AI cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

We present a new human-human dialogue dataset - PhotoChat, the first dataset that casts light on the photo sharing behavior in onlin emessaging. PhotoChat contains 12k dialogues, each of which is paired with a user photo that is shared during the conversation. Based on this dataset, we propose two tasks to facilitate research on image-text modeling: a photo-sharing intent prediction task that predicts whether one intends to share a photo in the next conversation turn, and a photo retrieval task that retrieves the most relevant photo according to the dialogue context. In addition, for both tasks, we provide baseline models using the state-of-the-art models and report their benchmark performances. The best image retrieval model achieves 10.4% recall@1 (out of 1000 candidates) and the best photo intent prediction model achieves 58.1% F1 score, indicating that the dataset presents interesting yet challenging real-world problems. We are releasing PhotoChat to facilitate future research work among the community.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

20 of 31 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
8 more
  1. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  2. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  3. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  4. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  5. 2018Image Chat: Engaging Grounded Conversationscited by 77
  6. 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
  7. 2018End-to-End Retrieval in Continuous Spacecited by 46
  8. 2020Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCOcited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 12; 2025: 12; 2026: 2.

43 papers.

Sort:

By area:

  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  4. 2021Multimodal Dialogue Response Generationcited by 32
  5. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  6. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  7. 2023A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Modelscited by 23
  8. 2024STICKERCONV: Generating Multimodal Empathetic Responses from Scratchcited by 22
  9. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  10. 2022DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Datasetcited by 17
  11. 2024UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generationcited by 17
  12. 2024MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasetscited by 10
31 more
  1. 2023EasyGen: Easing Multimodal Generation with BiDiffuser and LLMscited by 9
  2. 2024Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understandingcited by 9
  3. 2024Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledgecited by 8
  4. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8
  5. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7
  6. 2022Towards Building an Open-Domain Dialogue System Incorporated with Internet Memescited by 5
  7. 2023Large Language Models can Share Images, Too!cited by 5
  8. 2024Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Modelcited by 3
  9. 2025MuDoC: An Interactive Multimodal Document-grounded Conversational AI Systemcited by 3
  10. 2025Towards a Multimodal Document-grounded Conversational AI System for Educationcited by 3
  11. 2025ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversationscited by 3
  12. 2025Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactionscited by 3
  13. 2025GeoExplain: Multimodal Reasoning based on Hierarchy of Visual Information in Street Viewcited by 3
  14. 2025MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarizationcited by 3
  15. 2023Building Multimodal AI Chatbotscited by 2
  16. 2024DialCLIP: Empowering CLIP as Multi-Modal Dialog Retrievercited by 2
  17. 2022Resolving the Human Subjects Status of Machine Learning's Crowdworkerscited by 1
  18. 2024Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Modelscited by 1
  19. 2024An End-to-End Model for Photo-Sharing Multi-modal Dialogue Generationcited by 1
  20. 2025KwaiChat: A Large-Scale Video-Driven Multilingual Mixed-Type Dialogue Corpuscited by 1
  21. 2023VDialogUE: A Unified Evaluation Benchmark for Visually-grounded Dialoguecited by 0
  22. 2023Teaching Text-to-Image Models to Communicate in Dialogcited by 0
  23. 2023EXMODD: An EXplanatory Multimodal Open-Domain Dialogue datasetcited by 0
  24. 2024DIVE: Towards Descriptive and Diverse Visual Commonsense Generationcited by 0
  25. 2025MTPChat: A Multimodal Time-Aware Persona Dataset for Conversational Agentscited by 0
  26. 2025On the Effectiveness of Integration Methods for Multimodal Dialogue Response Retrievalcited by 0
  27. 2025MemeCMD: An Automatically Generated Chinese Multi-turn Dialogue Dataset with Contextually Retrieved Memescited by 0
  28. 2025F2RVLM: Boosting Fine-grained Fragment Retrieval for Multi-Modal Long-form Dialogue with Vision Language Modelcited by 0
  29. 2025Image-Seeking Intent Prediction for Cross-Device Product Searchcited by 0
  30. 2026Fine-grained Fragment Retrieval in Multi-modal Long-form Dialoguescited by 0
  31. 2026Beyond Local Surprise: Grounded Dialogue as Selective Belief Revision under Referential Uncertaintycited by 0