mlsanityData to 14 Oct 2026

AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models

arXiv 2509.12715 ↗First posted 16 Sep 2025cs.CV cs.ROcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated impressive performance on multimodal tasks through scaled architectures and extensive training. However, existing Mixture of Experts (MoE) approaches face challenges due to the asymmetry between visual and linguistic processing. Visual information is spatially complete, while language requires maintaining sequential context. As a result, MoE models struggle to balance modality-specific features and cross-modal interactions. Through systematic analysis, we observe that language experts in deeper layers progressively lose contextual grounding and rely more on parametric knowledge rather than utilizing the provided visual and linguistic information. To address this, we propose AsyMoE, a novel architecture that models this asymmetry using three specialized expert groups. We design intra-modality experts for modality-specific processing, hyperbolic inter-modality experts for hierarchical cross-modal interactions, and evidence-priority language experts to suppress parametric biases and maintain contextual grounding. Extensive experiments demonstrate that AsyMoE achieves 26.58% and 15.45% accuracy improvements over vanilla MoE and modality-specific MoE respectively, with 25.45% fewer activated parameters than dense models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 52 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

44 of 52 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models13 Aug
  2. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  3. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment28 May
  4. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  5. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  6. 2024LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs29 Jan
  7. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
3 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  3. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 29

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
17 more
  1. 2016A Diagram Is Worth A Dozen Imagescited by 804
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  3. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  4. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  6. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  8. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  9. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  10. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  11. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 62
  12. 2024COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Modelscited by 45
  13. 2024MouSi: Poly-Visual-Expert Vision-Language Modelscited by 24
  14. 2025HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learningcited by 17
  15. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  16. 2025Galaxy Walker: Geometry-aware VLMs For Galaxy-scale Understandingcited by 4
  17. 2025Enhancing Multi-modal Models with Heterogeneous MoE Adapters for Fine-tuningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfercited by 2
  2. 2026ReconMIL: Synergizing Latent Space Reconstruction with Bi-Stream Mamba for Whole Slide Image Analysiscited by 0
  3. 2026Scaling Laws for Multimodal Data Mixturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Models
  2. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
  3. 2025Astrea: A MOE-based Visual Understanding Model with Progressive Alignment
  4. 2026VEQ: Modality-Adaptive Quantization for MoE Vision-Language Models
  5. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate
  6. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
  7. 2026MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Models
  8. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective