mlsanityData to 7 Oct 2026

Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs

arXiv 2011.15124 ↗First posted 30 Nov 2020cs.CL cs.CVcited by 71

Not on the map. Why?

Abstract (arXiv)

Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of methods have been proposed for pretraining vision and language BERTs to tackle challenges at the intersection of these two key areas of AI. These models can be categorised into either single-stream or dual-stream encoders. We study the differences between these two categories, and show how they can be unified under a single theoretical framework. We then conduct controlled experiments to discern the empirical differences between five V&L BERTs. Our experiments show that training data and hyperparameters are responsible for most of the differences between the reported results, but they also reveal that the embedding layer plays a crucial role in these massive models.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 21; 2023: 22; 2024: 14; 2025: 0; 2026: 3.

71 papers.

Sort:

By area:

  1. 2022Visual Spatial Reasoningcited by 288
  2. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  3. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  4. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  5. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  6. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  7. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  8. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  9. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  10. 2022Kernelized Concept Erasurecited by 47
  11. 2022FashionViL: Fashion-Focused Vision-and-Language Representation Learningcited by 32
  12. 2023Zero-shot Composed Text-Image Retrievalcited by 31
38 more
  1. 2023GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Modelscited by 31
  2. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Modelscited by 28
  3. 2021Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrievalcited by 27
  4. 2022Image Retrieval from Contextual Descriptionscited by 26
  5. 2021Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answeringcited by 25
  6. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  7. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  8. 2022LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translationcited by 21
  9. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  10. 2022Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalizationcited by 20
  11. 2021VisQA: X-raying Vision and Language Reasoning in Transformerscited by 19
  12. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  13. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignmentcited by 17
  14. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16
  15. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  16. 2024How and where does CLIP process negation?cited by 16
  17. 2024Do Language Models Understand Time?cited by 14
  18. 20233D Feature Prediction for Masked-AutoEncoder-Based Point Cloud Pretrainingcited by 13
  19. 2023A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Cluescited by 11
  20. 2024Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNAcited by 11
  21. 2023A Study of Autoregressive Decoders for Multi-Tasking in Computer Visioncited by 10
  22. 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?cited by 8
  23. 2023The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Modelscited by 8
  24. 2021MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognitioncited by 7
  25. 2022Finding Structural Knowledge in Multimodal-BERTcited by 7
  26. 2022Learning by Hallucinating: Vision-Language Pre-training with Weak Supervisioncited by 6
  27. 2023A Two-Sided Discussion of Preregistration of NLP Researchcited by 6
  28. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  29. 2023Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Modelscited by 6
  30. 2023Multimodal Inverse Cloze Task for Knowledge-based Visual Question Answeringcited by 5
  31. 2023Vision Language Transformers: A Surveycited by 5
  32. 2024GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanationscited by 5
  33. 2021TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoningcited by 4
  34. 2022Grounding Commands for Autonomous Vehicles via Layer Fusion with Region-specific Dynamic Layer Attentioncited by 4
  35. 2022How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?cited by 4
  36. 2023A Survey on Image-text Multimodal Modelscited by 4
  37. 2024Acquiring Linguistic Knowledge from Multimodal Inputcited by 3
  38. 2022Training Vision-Language Models with Less Bimodal Supervisioncited by 2

50 most cited of 71 All 71 (JSON)