mlsanityData to 14 Oct 2026

SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs

arXiv 2604.23996 ↗First posted 27 Apr 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 60 of its 68 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

60 of 68 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models13 Aug
  2. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model2 Jul
  3. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities6 Jun
  4. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models28 May
  5. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  6. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training10 Oct
  7. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models17 Jul
  8. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model28 Jun
  9. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May
  10. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  11. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
2 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
34 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  7. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  11. 2025Muon is Scalable for LLM Trainingcited by 428
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  13. 2025Kimi-VL Technical Reportcited by 388
  14. 2025Seed1.5-VL Technical Reportcited by 373
  15. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  16. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  17. 2023From Sparse to Soft Mixtures of Expertscited by 248
  18. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  19. 2024Mixture of LoRA Expertscited by 181
  20. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 163
  21. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  22. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  23. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  24. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  25. 2025Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decodingcited by 55
  26. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 25
  27. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 20
  28. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 18
  29. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  30. 2025Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Expertscited by 15
  31. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
  32. 2023An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Trainingcited by 6
  33. 2025CoMoE: Contrastive Representation for Mixture-of-Experts in Parameter-Efficient Fine-tuningcited by 4
  34. 2025Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchronycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Structuring MoE Expert Selection for Agentic Reinforcement Learningcited by 0
  2. 2026Overcoming Data Imbalances with Expert Selection Biasescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
  2. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models
  3. 2026Information Router for Mitigating Modality Dominance in Vision-Language Models
  4. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
  5. 2026Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
  6. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping
  7. 2026MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
  8. 2023Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts