mlsanityData to 14 Oct 2026

GuessWhat?! Visual object discovery through multi-modal dialogue

arXiv 1611.08481 ↗First posted 23 Nov 2016cs.AI cs.CVcited by 113

Not on the map. Why?

Abstract (arXiv)

We introduce GuessWhat?!, a two-player guessing game as a testbed for research on the interplay of computer vision and dialogue systems. The goal of the game is to locate an unknown object in a rich image scene by asking a sequence of questions. Higher-level image understanding, like spatial reasoning and language grounding, is required to solve the proposed task. Our key contribution is the collection of a large-scale dataset consisting of 150K human-played games with a total of 800K visual question-answer pairs on 66K images. We explain our design decisions in collecting the dataset and introduce the oracle and questioner tasks that are associated with the two players of the game. We prototyped deep learning models to establish initial baselines of the introduced tasks.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 37; 2022: 25; 2023: 25; 2024: 10; 2025: 8; 2026: 8.

113 papers.

Sort:

By area:

  1. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  2. 2022RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Datacited by 123
  3. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 90
  4. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  5. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  6. 2021SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversationscited by 56
  7. 2024Toward Visual Grounding: A Surveycited by 52
  8. 2021Grounding 'Grounding' in NLPcited by 44
  9. 2021Language Grounding with 3D Objectscited by 43
  10. 2021YouRefIt: Embodied Reference Understanding with Language and Gesturecited by 33
  11. 2025Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challengescited by 31
  12. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 31
38 more
  1. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  3. 2022Image Retrieval from Contextual Descriptionscited by 26
  4. 2022FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulationcited by 24
  5. 2023Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Gamescited by 24
  6. 2024ChatShop: Interactive Information Seeking with Language Agentscited by 24
  7. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  8. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22
  9. 2025Referring to Any Personcited by 21
  10. 2021Open-domain clarification question generation without question examplescited by 20
  11. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 19
  12. 2021Structured Co-reference Graph Attention for Video-grounded Dialoguecited by 17
  13. 2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 17
  14. 2021OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contextscited by 16
  15. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  16. 2022Understanding Embodied Reference with Touch-Line Transformercited by 16
  17. 2021Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigationcited by 15
  18. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challengescited by 14
  19. 2021DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialoguecited by 13
  20. 2021A recipe for annotating grounded clarificationscited by 13
  21. 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
  22. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  23. 2023ScanERU: Interactive 3D Visual Grounding based on Embodied Reference Understandingcited by 12
  24. 2023Learning to Generate Equitable Text in Dialogue from Biased Training Datacited by 12
  25. 2021Channel Exchanging Networks for Multimodal and Multitask Dense Image Predictioncited by 11
  26. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  27. 2023VSTAR: A Video-grounded Dialogue Dataset for Situated Semantic Understanding with Scene and Topic Transitionscited by 11
  28. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  29. 2021Maintaining Common Ground in Dynamic Environmentscited by 9
  30. 2023Dealing with Semantic Underspecification in Multimodal NLPcited by 9
  31. 2021Learning Better Visual Dialog Agents with Pretrained Visual-Linguistic Representationcited by 8
  32. 2021Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialoguecited by 8
  33. 2022Co-VQA : Answering by Interactive Sub Question Sequencecited by 8
  34. 2023WildRefer: 3D Object Localization in Large-scale Dynamic Scenes with Multi-modal Visual Data and Natural Languagecited by 8
  35. 2024ActionVOS: Actions as Prompts for Video Object Segmentationcited by 8
  36. 2021Target-dependent UNITER: A Transformer-Based Multimodal Language Comprehension Model for Domestic Service Robotscited by 7
  37. 2022The Dialog Must Go On: Improving Visual Dialog via Generative Self-Trainingcited by 7
  38. 2023TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real Worldcited by 7

50 most cited of 113 All 113 (JSON)