mlsanityData to 7 Oct 2026

An Empirical Study of Training End-to-End Vision-and-Language Transformers

arXiv 2111.02387 ↗First posted 3 Nov 2021cs.CV cs.CL cs.LGcited by 271

Not on the map. Why?

Cited by 97 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be more efficient than previous region-feature-based methods, their performance on downstream tasks often degrades significantly. In this paper, we present METER, a Multimodal End-to-end TransformER framework, through which we investigate how to design and pre-train a fully transformer-based VL model in an end-to-end manner. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion module (e.g., merged attention vs. co-attention), architectural design (e.g., encoder-only vs. encoder-decoder), and pre-training objectives (e.g., masked image modeling). We conduct comprehensive experiments and provide insights on how to train a performant VL transformer. METER achieves an accuracy of 77.64% on the VQAv2 test-std set using only 4M images for pre-training, surpassing the state-of-the-art region-feature-based model by 1.04%, and outperforming the previous best fully transformer-based model by 1.6%. Notably, when further scaled up, our best VQA model achieves an accuracy of 80.54%. Code and pre-trained models are released at https://github.com/zdou0830/METER.

Key facts. Cited by 271 papers in the map (data through 7 Oct 2026). Builds on 54 of its 67 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

54 of 67 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  13. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  14. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  15. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  16. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  17. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  18. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  19. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  20. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  24. 2021Going deeper with Image Transformerscited by 759
  25. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  26. 2019Text Summarization with Pretrained Encoderscited by 553
  27. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  28. 2017Deep Layer Aggregationcited by 436
  29. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  30. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  31. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  32. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  33. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  34. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  35. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  36. 2016Exploring the Limits of Language Modelingcited by 172
  37. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  38. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 65; 2023: 111; 2024: 57; 2025: 24; 2026: 9.

271 papers.

Sort:

By area:

  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  7. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  8. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  9. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
  10. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  11. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  12. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
38 more
  1. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  2. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  3. 2023A Cookbook of Self-Supervised Learningcited by 248
  4. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  5. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 210
  6. 2023Multimodal Large Language Models: A Surveycited by 193
  7. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  8. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  11. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  12. 2023Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrievalcited by 131
  13. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  14. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  15. 2024Vision-Language Models for Medical Report Generation and Visual Question Answering: A Reviewcited by 121
  16. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  17. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  18. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  19. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  20. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  21. 2023Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Modelscited by 106
  22. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  23. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  24. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  25. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  26. 2023InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discoverycited by 86
  27. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  28. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  29. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  30. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  31. 2022PLOT: Prompt Learning with Optimal Transport for Vision-Language Modelscited by 64
  32. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  33. 2022Your ViT is Secretly a Hybrid Discriminative-Generative Diffusion Modelcited by 58
  34. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 57
  35. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  36. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  37. 2023When Brain-inspired AI Meets AGIcited by 54
  38. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53

50 most cited of 271 All 271 (JSON)