mlsanityData to 7 Oct 2026

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

arXiv 2405.11273 ↗First posted 18 May 2024cs.AI cs.CL cs.CVcited by 107

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at https://github.com/HITsz-TMG/UMOE-Scaling-Unified-Multimodal-LLMs.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 86 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

65 of 86 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  10. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  11. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration15 Jun
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
1 more
  1. 2023LMEye: An Interactive Perception Network for Large Language Models5 May

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2023Qwen Technical Reportcited by 3,273
38 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2024Mixtral of Expertscited by 1,785
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  13. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  15. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  17. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  18. 2020A Survey on Visual Transformercited by 653
  19. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  20. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  21. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  22. 2022Multimodal Learning with Transformers: A Surveycited by 399
  23. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  24. 2019Clotho: An Audio Captioning Datasetcited by 354
  25. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  26. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  27. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  28. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  29. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  30. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  31. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  32. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  33. 2023AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Modelcited by 93
  34. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 84
  35. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  36. 2023Prefix tuning for automated audio captioningcited by 26
  37. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  38. 2022Rethinking Collaborative Metric Learning: Toward an Efficient Alternative without Negative Samplingcited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 47; 2026: 46.

107 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  3. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  4. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  5. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  6. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  7. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  8. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  9. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  10. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 38
  11. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Expertscited by 36
  12. 2024DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identificationcited by 29
38 more
  1. 2025FilmAgent: A Multi-Agent Framework for End-to-End Film Automation in Virtual 3D Spacescited by 28
  2. 2025MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoderscited by 25
  3. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  4. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  5. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  6. 2025MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Expertscited by 19
  7. 2025AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generationcited by 17
  8. 2025TimeExpert: An Expert-Guided Video LLM for Video Temporal Groundingcited by 14
  9. 2025SAIL-VL2 Technical Reportcited by 14
  10. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  11. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  12. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  13. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 9
  14. 2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
  15. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  16. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMscited by 6
  17. 2025Route Experts by Sequence, not by Tokencited by 5
  18. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decodingcited by 5
  19. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  20. 2025VideoVista-CulturalLingo: 360^circ Horizons-Bridging Cultures, Languages, and Domains in Video Comprehensioncited by 4
  21. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  22. 2025GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Modelscited by 4
  23. 2025Towards Accurate and Efficient 3D Object Detection for Autonomous Driving: A Mixture of Experts Computing System on Edgecited by 4
  24. 2025UniMoE-Audio: Unified Speech and Music Generation with Dynamic-Capacity MoEcited by 4
  25. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Freecited by 4
  26. 2024An Adaptive Framework for Generating Systematic Explanatory Answer in Online Q&A Platformscited by 3
  27. 2025Exploiting Mixture-of-Experts Redundancy Unlocks Multimodal Generative Abilitiescited by 3
  28. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  29. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
  30. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilitiescited by 3
  31. 2025End-to-end Listen, Look, Speak and Actcited by 3
  32. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Expertscited by 3
  33. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 3
  34. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  35. 2025Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agentscited by 2
  36. 2025Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilitiescited by 2
  37. 2025Farm-LightSeek: An Edge-centric Multimodal Agricultural IoT Data Analytics Framework with Lightweight LLMscited by 2
  38. 2025Multimodal LLM With Hierarchical Mixture-of-Experts for VQA on 3D Brain MRIcited by 2

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  2. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  3. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  4. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
  5. 2026MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
  6. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
  7. 2024MoExtend: Tuning New Experts for Modality and Task Extension
  8. 2024SoupLM: Model Integration in Large Language and Multi-Modal Models