mlsanityData to 7 Oct 2026

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

arXiv 2401.15947 ↗First posted 29 Jan 2024cs.CVcited by 298

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Recent advances demonstrate that scaling Large Vision-Language Models (LVLMs) effectively improves downstream task performances. However, existing scaling methods enable all model parameters to be active for each token in the calculation, which brings massive training and inferring costs. In this work, we propose a simple yet effective training strategy MoE-Tuning for LVLMs. This strategy innovatively addresses the common issue of performance degradation in multi-modal sparsity learning, consequently constructing a sparse model with an outrageous number of parameters but a constant computational cost. Furthermore, we present the MoE-LLaVA, a MoE-based sparse LVLM architecture, which uniquely activates only the top-k experts through routers during deployment, keeping the remaining experts inactive. Extensive experiments show the significant performance of MoE-LLaVA in a variety of visual understanding and object hallucination benchmarks. Remarkably, with only approximately 3B sparsely activated parameters, MoE-LLaVA demonstrates performance comparable to the LLaVA-1.5-7B on various visual understanding datasets and even surpasses the LLaVA-1.5-13B in object hallucination benchmark. Through MoE-LLaVA, we aim to establish a baseline for sparse LVLMs and provide valuable insights for future research in developing more efficient and effective multi-modal learning systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.

Key facts. Cited by 298 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 87 of its 116 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

87 of 116 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec
  4. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  5. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  8. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  11. 2023A Survey on Multimodal Large Language Models23 Jun
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
3 more
  1. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 72

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023Mistral 7Bcited by 3,065
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  15. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  16. 2023PaLM 2 Technical Reportcited by 1,154
  17. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  18. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  19. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  21. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  22. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  23. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  24. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  25. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  26. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  27. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  28. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  29. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  30. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  31. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  32. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  33. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  34. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  35. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  36. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  37. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  38. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 116; 2025: 120; 2026: 62.

298 papers.

Sort:

By area:

  1. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  3. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  4. 2024What matters when building vision-language models?cited by 321
  5. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  7. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  8. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  9. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  10. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  11. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 169
38 more
  1. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  2. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  3. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  4. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  5. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  6. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  7. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  8. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  9. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  10. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  11. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  12. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  13. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  14. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  15. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  16. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  17. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  18. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  19. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  20. 2024Needle In A Multimodal Haystackcited by 56
  21. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  22. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Modelscited by 55
  23. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  24. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  25. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  26. 2024Theory on Mixture-of-Experts in Continual Learningcited by 50
  27. 2024CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMscited by 49
  28. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  29. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  30. 2025Scaling Laws for Native Multimodal Modelscited by 47
  31. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Modelscited by 45
  32. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  33. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  34. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  35. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  36. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  37. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  38. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37

50 most cited of 298 All 298 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EffiVLM-BENCH: A Comprehensive Benchmark for Evaluating Training-Free Acceleration in Large Vision-Language Models
  2. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
  3. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
  4. 2025Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models
  5. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
  6. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models
  7. 2026When LLaVA Meets Objects: Token Composition for Vision-Language-Models
  8. 2025Rethinking Visual Information Processing in Multimodal LLMs