mlsanityData to 7 Oct 2026

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

arXiv 2405.05949 ↗First posted 9 May 2024cs.CVcited by 56

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (LLMs) have focused primarily on scaling by increasing text-image pair data and enhancing LLMs to improve performance on multimodal tasks. However, these scaling approaches are computationally expensive and overlook the significance of improving model capabilities from the vision side. Inspired by the successful applications of Mixture-of-Experts (MoE) in LLMs, which improves model scalability during training while keeping inference costs similar to those of smaller models, we propose CuMo. CuMo incorporates Co-upcycled Top-K sparsely-gated Mixture-of-experts blocks into both the vision encoder and the MLP connector, thereby enhancing the multimodal LLMs with minimal additional activated parameters during inference. CuMo first pre-trains the MLP blocks and then initializes each expert in the MoE block from the pre-trained MLP block during the visual instruction tuning stage. Auxiliary losses are used to ensure a balanced loading of experts. CuMo outperforms state-of-the-art multimodal LLMs across various VQA and visual-instruction-following benchmarks using models within each model size group, all while training exclusively on open-sourced datasets. The code and model weights for CuMo are open-sourced at https://github.com/SHI-Labs/CuMo.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 75 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

59 of 75 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  2. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  3. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  5. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models21 Dec
  6. 2023VILA: On Pre-training for Visual Language Models12 Dec
  7. 2023Honeybee: Locality-enhanced Projector for Multimodal LLM11 Dec
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
3 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2023Mistral 7Bcited by 3,065
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
32 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  8. 2016A Diagram Is Worth A Dozen Imagescited by 793
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  12. 2021InfographicVQAcited by 430
  13. 2021OCR-free Document Understanding Transformercited by 400
  14. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  15. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  16. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  17. 2024What matters when building vision-language models?cited by 321
  18. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  19. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  20. 2022Neighborhood Attention Transformercited by 235
  21. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  22. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  23. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  24. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  25. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  26. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  27. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  28. 2024When Do We Not Need Larger Vision Models?cited by 65
  29. 2022Dilated Neighborhood Attention Transformercited by 64
  30. 2021Document Collection Visual Question Answeringcited by 45
  31. 2022Residual Mixture of Expertscited by 39
  32. 2022VMFormer: End-to-End Video Matting with Transformercited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 27; 2026: 11.

56 papers.

Sort:

By area:

  1. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  2. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
  3. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  4. 2024Dense Connector for MLLMscited by 48
  5. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 44
  6. 2024CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Modelscited by 37
  7. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  8. 2024Wings: Learning Multimodal LLMs without Text-only Forgettingcited by 23
  9. 2024EVLM: An Efficient Vision-Language Model for Visual Understandingcited by 23
  10. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  11. 2024Automate Strategy Finding with LLM in Quant Investmentcited by 21
  12. 2025Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinementcited by 18
38 more
  1. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  2. 2024YourMT3+: Multi-instrument Music Transcription with Enhanced Transformer Architectures and Cross-dataset Stem Augmentationcited by 13
  3. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  4. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  5. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  6. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  7. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  8. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  9. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  10. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  11. 2025TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understandingcited by 6
  12. 2024Measuring Agreeableness Bias in Multimodal Modelscited by 4
  13. 2025Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edgecited by 4
  14. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Modelscited by 4
  15. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  16. 2025Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Expertscited by 3
  17. 2025Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentationcited by 3
  18. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  19. 2025MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbationscited by 2
  20. 2025AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Modelscited by 2
  21. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRIcited by 2
  22. 2025Metacognitive Sensitivity for Test-Time Dynamic Model Selectioncited by 2
  23. 2026Aligned Multi-View Scripts for Universal Chart-to-Code Generationcited by 2
  24. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  25. 2025D-Attn: Decomposed Attention for Large Vision-and-Language Modelscited by 1
  26. 2025Short-video Propagation Influence Rating: A New Real-world Dataset and A New Large Graph Modelcited by 1
  27. 2025WAVE++: Capturing Within-Task Variance for Continual Relation Extraction with Adaptive Promptingcited by 1
  28. 2025MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Modelscited by 1
  29. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoEcited by 1
  30. 2026DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoEcited by 1
  31. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  32. 2024Gamified crowd-sourcing of high-quality data for visual fine-tuningcited by 0
  33. 2024Multimodal Instruction Tuning with Hybrid State Space Modelscited by 0
  34. 2025Machine Learning and statistical classification of CRISPR-Cas12a diagnostic assayscited by 0
  35. 2025Improving vision-language alignment with graph spiking hybrid Networkscited by 0
  36. 2025ISALux: Illumination and Segmentation Aware Transformer Employing Mixture of Experts for Low Light Image Enhancementcited by 0
  37. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  38. 2026CARPE: Context-Aware Image Representation Prioritization via Ensemble for Large Vision-Language Modelscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  2. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
  3. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  4. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
  5. 2023Scaling Vision-Language Models with Sparse Mixture of Experts
  6. 2023On the Performance of Multimodal Language Models
  7. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models
  8. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints