mlsanityData to 14 Oct 2026

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

arXiv 2111.02358 ↗First posted 3 Nov 2021cs.CV cs.CL cs.LGcited by 425

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We present a unified Vision-Language pretrained Model (VLMo) that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Specifically, we introduce Mixture-of-Modality-Experts (MoME) Transformer, where each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of MoME, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. Moreover, we propose a stagewise pre-training strategy, which effectively leverages large-scale image-only and text-only data besides image-text pairs. Experimental results show that VLMo achieves state-of-the-art results on various vision-language tasks, including VQA, NLVR2 and image-text retrieval. The code and pretrained models are available at https://aka.ms/vlmo.

Key facts. Cited by 425 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 65 references that are in the map. First posted on arXiv 3 Nov 2021.

Builds on

45 of 65 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  2. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
31 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  14. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  16. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  17. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  18. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  19. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  20. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  21. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  22. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  23. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  24. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  25. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  26. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  27. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  28. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  29. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  30. 2019Cross-Lingual Natural Language Generation via Pre-Trainingcited by 63
  31. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 71; 2023: 134; 2024: 111; 2025: 81; 2026: 26.

425 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  9. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  10. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  11. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  12. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
38 more
  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023Learning to Exploit Temporal Structure for Biomedical Vision-Language Processingcited by 211
  6. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  7. 2023Multimodal Large Language Models: A Surveycited by 195
  8. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  9. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 187
  10. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  11. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  12. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  13. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  14. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  15. 2023Emu: Generative Pretraining in Multimodalitycited by 157
  16. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  17. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  18. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 151
  19. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 145
  20. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  21. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  22. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  23. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  24. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  25. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  26. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 126
  27. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  28. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 121
  29. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 121
  30. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  31. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  32. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  33. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  34. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  35. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
  36. 2023SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understandingcited by 111
  37. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  38. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109

50 most cited of 425 All 425 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Encoder-Free Vision-Language Models
  2. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
  3. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  4. 2026CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning
  5. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  6. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  7. 2025Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
  8. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels