mlsanityData to 7 Oct 2026

Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers

arXiv 2102.00529 ↗First posted 31 Jan 2021cs.CL cs.CVcited by 63

Not on the map. Why?

Abstract (arXiv)

Recently multimodal transformer models have gained popularity because their performance on language and vision tasks suggest they learn rich visual-linguistic representations. Focusing on zero-shot image retrieval tasks, we study three important factors which can impact the quality of learned representations: pretraining data, the attention mechanism, and loss functions. By pretraining models on six datasets, we observe that dataset noise and language similarity to our downstream task are important indicators of model performance. Through architectural analysis, we learn that models with a multimodal attention mechanism can outperform deeper models with modality specific attention mechanisms. Finally, we show that successful contrastive losses used in the self-supervised learning literature do not yield similar performance gains when used in multimodal transformers

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 31 Jan 2021.

Builds on

39 of 52 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
27 more
  1. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  2. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  3. 2019Contrastive Multiview Codingcited by 1,297
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  6. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2019Designing and Interpreting Probes with Control Taskscited by 608
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  13. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  14. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  15. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  16. 2019Connecting Vision and Language with Localized Narrativescited by 225
  17. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  18. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  19. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  20. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  21. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  22. 2017Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challengecited by 76
  23. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  24. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27
  25. 2020Are we pretraining it right? Digging deeper into visio-linguistic pretrainingcited by 25
  26. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17
  27. 2018Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrievalcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 20; 2023: 17; 2024: 8; 2025: 6; 2026: 1.

63 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2023ClimaX: A foundation model for weather and climatecited by 325
  5. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  6. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  7. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  8. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  9. 2023Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrievalcited by 131
  10. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  11. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  12. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
38 more
  1. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  2. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  5. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  6. 2021MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognitioncited by 44
  7. 2023Transformers in Speech Processing: A Surveycited by 38
  8. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  9. 2022Contrastive Audio-Language Learning for Musiccited by 32
  10. 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
  11. 2023Brain encoding models based on multimodal transformers can transfer across language and visioncited by 31
  12. 2023Multimodal Graph Learning for Generative Taskscited by 27
  13. 2021Sliced Recursive Transformercited by 26
  14. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  15. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  16. 2022Evaluating Self-Supervised Learning for Molecular Graph Embeddingscited by 22
  17. 2021Distilled Dual-Encoder Model for Vision-Language Understandingcited by 21
  18. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignmentcited by 19
  19. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  20. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  21. 2022AutoTransition: Learning to Recommend Video Transition Effectscited by 13
  22. 2021Telling the What while Pointing to the Where: Multimodal Queries for Image Retrievalcited by 12
  23. 2025InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactionscited by 7
  24. 2025Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Datacited by 7
  25. 2021MLP Architectures for Vision-and-Language Modeling: An Empirical Studycited by 6
  26. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  27. 2023SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Imagescited by 5
  28. 2024ChitroJera: A Regionally Relevant Visual Question Answering Dataset for Banglacited by 5
  29. 2023MEGA: Multimodal Alignment Aggregation and Distillation For Cinematic Video Segmentationcited by 4
  30. 2024Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classificationcited by 4
  31. 2023Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Promptscited by 3
  32. 2022Vision-and-Language Pretrainingcited by 2
  33. 2022Training Vision-Language Models with Less Bimodal Supervisioncited by 2
  34. 2022Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptionscited by 2
  35. 2022Compound Tokens: Channel Fusion for Vision-Language Representation Learningcited by 2
  36. 2023Read, Look or Listen? What's Needed for Solving a Multimodal Datasetcited by 2
  37. 2024Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Mediacited by 2
  38. 2022One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Taskscited by 1

50 most cited of 63 All 63 (JSON)