mlsanityData to 7 Oct 2026

Visual Dialog

arXiv 1611.08669 ↗First posted 26 Nov 2016cs.CV cs.AI cs.CLcited by 377

Not on the map. Why?

Cited by 83 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialog with humans in natural, conversational language about visual content. Specifically, given an image, a dialog history, and a question about the image, the agent has to ground the question in image, infer context from history, and answer the question accurately. Visual Dialog is disentangled enough from a specific downstream task so as to serve as a general test of machine intelligence, while being grounded in vision enough to allow objective evaluation of individual responses and benchmark progress. We develop a novel two-person chat data-collection protocol to curate a large-scale Visual Dialog dataset (VisDial). VisDial v0.9 has been released and contains 1 dialog with 10 question-answer pairs on ~120k images from COCO, with a total of ~1.2M dialog question-answer pairs. We introduce a family of neural encoder-decoder models for Visual Dialog with 3 encoders -- Late Fusion, Hierarchical Recurrent Encoder and Memory Network -- and 2 decoders (generative and discriminative), which outperform a number of sophisticated baselines. We propose a retrieval-based evaluation protocol for Visual Dialog where the AI agent is asked to sort a set of candidate answers and evaluated on metrics such as mean-reciprocal-rank of human response. We quantify gap between machine and human performance on the Visual Dialog task via human studies. Putting it all together, we demonstrate the first 'visual chatbot'! Our dataset, code, trained models and visual chatbot are available on https://visualdialog.org

Key facts. Cited by 377 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 69; 2022: 81; 2023: 71; 2024: 71; 2025: 53; 2026: 31.

377 papers.

Sort:

By area:

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  7. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  10. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  12. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
38 more
  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  2. 2023Generating Images with Multimodal Language Modelscited by 259
  3. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  5. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  6. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  7. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  8. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
  9. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  10. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  11. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  12. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  13. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  14. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  15. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  16. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  17. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  18. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  19. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  20. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  21. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  22. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  23. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  24. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  25. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  26. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  27. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  28. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  29. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  30. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  31. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  32. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  33. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  34. 2021Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systemscited by 73
  35. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 68
  36. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  37. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Datacited by 67
  38. 2023Grounding 3D Object Affordance from 2D Interactions in Imagescited by 62

50 most cited of 377 All 377 (JSON)