mlsanityData to 7 Oct 2026

Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts

arXiv 2206.02770 ↗First posted 6 Jun 2022cs.CVcited by 202

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Large sparsely-activated models have obtained excellent performance in multiple domains. However, such models are typically trained on a single modality at a time. We present the Language-Image MoE, LIMoE, a sparse mixture of experts model capable of multimodal learning. LIMoE accepts both images and text simultaneously, while being trained using a contrastive loss. MoEs are a natural fit for a multimodal backbone, since expert layers can learn an appropriate partitioning of modalities. However, new challenges arise; in particular, training stability and balanced expert utilization, for which we propose an entropy-based regularization scheme. Across multiple scales, we demonstrate remarkable performance improvement over dense models of equivalent computational cost. LIMoE-L/16 trained comparably to CLIP-L/14 achieves 78.6% zero-shot ImageNet accuracy (vs. 76.2%), and when further scaled to H/14 (with additional data) it achieves 84.1%, comparable to state-of-the-art methods which use larger custom per-modality backbones and pre-training schemes. We analyse the quantitative and qualitative behavior of LIMoE, and demonstrate phenomena such as differing treatment of the modalities and the organic emergence of modality-specific experts.

Key facts. Cited by 202 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 44 of its 54 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

44 of 54 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
32 more
  1. 2020Supervised Contrastive Learningcited by 2,803
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2021Scaling Vision Transformerscited by 993
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  15. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  16. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  17. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  18. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  19. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  20. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  21. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  22. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  23. 2021Hash Layers For Large Sparse Modelscited by 236
  24. 2019Fixing the train-test resolution discrepancycited by 209
  25. 2020Characterising Bias in Compressed Modelscited by 162
  26. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  27. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  28. 2021UFO: A UniFied TransfOrmer for Vision-Language Representation Learningcited by 45
  29. 2021Cross-token Modeling with Conditional Computationcited by 17
  30. 2020Routing Networks with Co-training for Continual Learningcited by 14
  31. 2021Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Textcited by 14
  32. 2022A Comprehensive Study of Vision Transformers on Dense Prediction Taskscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 43; 2024: 59; 2025: 52; 2026: 32.

202 papers.

Sort:

By area:

  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2023From Sparse to Soft Mixtures of Expertscited by 245
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  6. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  7. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  8. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  9. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  10. 2022FlexiViT: One Model for All Patch Sizescited by 151
  11. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  12. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
38 more
  1. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  2. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  3. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  4. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  5. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  6. 2025ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulationcited by 87
  7. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  8. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  9. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  10. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  11. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  12. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  13. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  14. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
  15. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61
  16. 2024Task-Customized Mixture of Adapters for General Image Fusioncited by 59
  17. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  18. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  19. 2023Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusioncited by 54
  20. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  21. 2025Scaling Laws for Native Multimodal Modelscited by 47
  22. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  23. 2025Advancing Expert Specialization for Better MoEcited by 37
  24. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  25. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  26. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  27. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  28. 2022Sparsity-Constrained Optimal Transportcited by 32
  29. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  30. 2023FOCAL: Contrastive Learning for Multimodal Time-Series Sensing Signals in Factorized Orthogonal Latent Spacecited by 31
  31. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 31
  32. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  33. 2025I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Expertscited by 29
  34. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorizationcited by 28
  35. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  36. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  37. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  38. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23

50 most cited of 202 All 202 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  2. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  3. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models
  6. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling
  7. 2026MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
  8. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping