mlsanityData to 7 Oct 2026

MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

arXiv 2407.12709 ↗First posted 17 Jul 2024cs.CVcited by 45

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have demonstrated impressive capabilities across various vision-language tasks. However, a generalist MLLM typically underperforms compared with a specialist MLLM on most VL tasks, which can be attributed to task interference. In this paper, we propose a mixture of multimodal experts (MoME) to mitigate task interference and obtain a generalist MLLM. Our MoME is composed of two key components, a mixture of vision experts (MoVE) and a mixture of language experts (MoLE). MoVE can adaptively modulate the features transformed from various vision encoders, and has a strong compatibility in transformation architecture. MoLE incorporates sparsely gated experts into LLMs to achieve painless improvements with roughly unchanged inference costs. In response to task interference, our MoME specializes in both vision and language modality to adapt to task discrepancies. Extensive experiments show that MoME significantly improves the performance of generalist MLLMs across various VL tasks. The source code is released at https://github.com/JiuTian-VL/MoME

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 64 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

55 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  2. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRA24 Feb
  3. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec
  7. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge20 Nov
  8. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  9. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models13 Oct
  10. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model8 Oct
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
31 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2024Mixtral of Expertscited by 1,785
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  12. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  13. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  15. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  16. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  17. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  18. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2021InfographicVQAcited by 430
  21. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  22. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  23. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  24. 2019Scene Text Visual Question Answeringcited by 327
  25. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  26. 2022Visual Spatial Reasoningcited by 288
  27. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  28. 2019Attentive Single-Tasking of Multiple Taskscited by 148
  29. 2023DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understandingcited by 106
  30. 2023Octavius: Mitigating Task Interference in MLLMs via LoRA-MoEcited by 25
  31. 2023Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitivescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 23; 2026: 17.

45 papers.

Sort:

By area:

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  2. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  3. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 83
  4. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 70
  5. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  6. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  7. 2024FoundIR: Unleashing Million-scale Training Data to Advance Foundation Models for Image Restorationcited by 37
  8. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 22
  9. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22
  10. 2025FALCON: Resolving Visual Redundancy and Fragmentation in High-resolution Multimodal Large Language Models via Visual Registerscited by 20
  11. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 18
  12. 2025Mixture of Experts in Large Language Modelscited by 15
33 more
  1. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  2. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  3. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  4. 2025Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptationcited by 9
  5. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 9
  6. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  7. 2024Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Modelscited by 7
  8. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  9. 2024Hints of Prompt: Enhancing Visual Representation for Multimodal LLMs in Autonomous Drivingcited by 6
  10. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  11. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 4
  12. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  13. 2025Vision Remember: Recovering Visual Information in Efficient LVLM with Vision Feature Resamplingcited by 2
  14. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Expertscited by 2
  15. 2026SplatWeaver: Learning to Allocate Gaussian Primitives for Generalizable Novel View Synthesiscited by 2
  16. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoEcited by 1
  17. 2025MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioningcited by 1
  18. 2026Learning to Accelerate Vision-Language-Action Models through Adaptive Visual Token Cachingcited by 1
  19. 2026Modeling Cross-vision Synergy for Unified Large Vision Modelcited by 1
  20. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  21. 2026AME-TS: Anchored Mixture-of-Experts for Time Series Forecastingcited by 1
  22. 2026BrainFIBRE: A Foundation Model via Information Decomposition for Brain Microstructurecited by 1
  23. 2026MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understandingcited by 1
  24. 2025Neural Inhibition Improves Dynamic Routing and Mixture of Expertscited by 0
  25. 2025Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Modelscited by 0
  26. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  27. 2026The Headless Firm: How AI Reshapes Enterprise Boundariescited by 0
  28. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 0
  29. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  30. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0
  31. 2026Spatio-Temporal Mixture-of-Modality-Experts Diffusion for Quantitative DCE-MRI Synthesis from Incomplete MR Sequencescited by 0
  32. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  33. 2026Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  2. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
  3. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  4. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
  7. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts
  8. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model