mlsanityData to 14 Oct 2026

Scaling Vision-Language Models with Sparse Mixture of Experts

arXiv 2303.07226 ↗First posted 13 Mar 2023cs.CV cs.CLcited by 88

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The field of natural language processing (NLP) has made significant strides in recent years, particularly in the development of large-scale vision-language models (VLMs). These models aim to bridge the gap between text and visual information, enabling a more comprehensive understanding of multimedia data. However, as these models become larger and more complex, they also become more challenging to train and deploy. One approach to addressing this challenge is the use of sparsely-gated mixture-of-experts (MoE) techniques, which divide the model into smaller, specialized sub-models that can jointly solve a task. In this paper, we explore the effectiveness of MoE in scaling vision-language models, demonstrating its potential to achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling VLMs. We hope our work will inspire further research into the use of MoE for scaling large-scale vision-language models and other multimodal machine learning applications.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 96 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

66 of 96 references.

References · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  13. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  14. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  16. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  18. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  19. 2016Deep Networks with Stochastic Depthcited by 979
  20. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  21. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  22. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  23. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  24. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  26. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  27. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  28. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  29. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  30. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  31. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  32. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  33. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  34. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  35. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  36. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  37. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 303
  38. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 36; 2025: 24; 2026: 13.

88 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  3. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  4. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  5. 2023RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Pathscited by 142
  6. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  7. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  8. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuningcited by 106
  9. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  10. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  11. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 68
  12. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
38 more
  1. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  2. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2025Scaling Laws for Native Multimodal Modelscited by 50
  5. 2023ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detectioncited by 48
  6. 2023OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllerscited by 47
  7. 2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
  8. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 45
  9. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  10. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  11. 2024Customizing Language Models with Instance-wise LoRA for Sequential Recommendationcited by 32
  12. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  13. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  14. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 25
  15. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  16. 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perceptioncited by 19
  17. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  18. 2025A Survey on Efficient Vision-Language Modelscited by 19
  19. 2023Unlocking Emergent Modularity in Large Language Modelscited by 17
  20. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16
  21. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
  22. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  23. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 14
  24. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  25. 2024UniRestorer: Universal Image Restoration via Adaptively Estimating Image Degradation at Proper Granularitycited by 14
  26. 2023Exploring Sparse MoE in GANs for Text-conditioned Image Synthesiscited by 13
  27. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 13
  28. 2025X-Fusion: Introducing New Modality to Frozen Large Language Modelscited by 13
  29. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  30. 2024Ensembling Diffusion Models via Adaptive Feature Aggregationcited by 12
  31. 2024Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Controlcited by 11
  32. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  33. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Expertscited by 11
  34. 2023DeepSpeed-VisualChat: Multi-Round Multi-Image Interleave Chat via Multi-Modal Causal Attentioncited by 10
  35. 2023Towards More Unified In-context Visual Understandingcited by 10
  36. 2024Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignmentcited by 10
  37. 2025Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detectioncited by 8
  38. 2024Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Modelscited by 7

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  2. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  3. 2025Mixture of Experts in Large Language Models
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding
  5. 2023Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
  6. 2025Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models
  7. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
  8. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models