mlsanityData to 7 Oct 2026

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

arXiv 1912.02379 ↗First posted 5 Dec 2019cs.LG cs.CL cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

Prior work in visual dialog has focused on training deep neural models on VisDial in isolation. Instead, we present an approach to leverage pretraining on related vision-language datasets before transferring to visual dialog. We adapt the recently proposed ViLBERT (Lu et al., 2019) model for multi-turn visually-grounded conversations. Our model is pretrained on the Conceptual Captions and Visual Question Answering datasets, and finetuned on VisDial. Our best single model outperforms prior published work (including model ensembles) by more than 1% absolute on NDCG and MRR. Next, we find that additional finetuning using "dense" annotations in VisDial leads to even higher NDCG -- more than 10% over our base model -- but hurts MRR -- more than 17% below our base model! This highlights a trade-off between the two primary metrics -- NDCG and MRR -- which we find is due to dense annotations not correlating well with the original ground-truth answers to questions.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 11; 2022: 24; 2023: 14; 2024: 6; 2025: 5.

60 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  6. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  7. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  8. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  9. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  10. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  11. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  12. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
38 more
  1. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  2. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  3. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  4. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  5. 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
  6. 2022Multimodal Knowledge Alignment with Reinforcement Learningcited by 30
  7. 2022Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spottingcited by 27
  8. 2023Chatting Makes Perfect: Chat-based Image Retrievalcited by 25
  9. 2022UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialogcited by 19
  10. 2021Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventionscited by 15
  11. 2021GoG: Relation-aware Graph-over-Graph Network for Visual Dialogcited by 13
  12. 2021Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generationcited by 11
  13. 2022Video Dialog as Conversation about Objects Living in Space-Timecited by 11
  14. 2021Learning to Ground Visual Objects for Visual Dialogcited by 10
  15. 2024Dynamic Transformer Architecture for Continual Learning of Multimodal Taskscited by 8
  16. 2022The Dialog Must Go On: Improving Visual Dialog via Generative Self-Trainingcited by 7
  17. 2022VD-PCR: Improving Visual Dialog with Pronoun Coreference Resolutioncited by 7
  18. 2023Modality-aware Transformer for Financial Time series Forecastingcited by 7
  19. 2025CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detectioncited by 6
  20. 2021Ensemble of MRR and NDCG models for Visual Dialogcited by 5
  21. 2021Saying the Unseen: Video Descriptions via Dialog Agentscited by 5
  22. 2021The JDDC 2.0 Corpus: A Large-Scale Multimodal Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Servicecited by 5
  23. 2022Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learningcited by 5
  24. 2022Transformer-based Localization from Embodied Dialog with Large-scale Pre-trainingcited by 5
  25. 2022End-to-End Multimodal Representation Learning for Video Dialogcited by 5
  26. 2022Unified Multimodal Model with Unlikelihood Training for Visual Dialogcited by 5
  27. 2023Improving Vision-and-Language Reasoning via Spatial Relations Modelingcited by 5
  28. 2023InfoVisDial: An Informative Visual Dialogue Dataset by Bridging Large Multimodal and Language Modelscited by 5
  29. 2024KG-TREAT: Pre-training for Treatment Effect Estimation by Synergizing Patient Data with Knowledge Graphscited by 5
  30. 2025Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrievalcited by 5
  31. 2022Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialogcited by 4
  32. 2022DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attentioncited by 4
  33. 2023Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversationscited by 4
  34. 2023mathbbVD-mathbbGR: Boosting mathbbVisual mathbbDialog with Cascaded Spatial-Temporal Multi-Modal mathbbGRaphscited by 4
  35. 2025RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systemscited by 4
  36. 2022Modeling Coreference Relations in Visual Dialogcited by 3
  37. 2023Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inferencecited by 3
  38. 2023Which way is `right'?: Uncovering limitations of Vision-and-Language Navigation modelcited by 3

50 most cited of 60 All 60 (JSON)