mlsanityData to 14 Oct 2026

SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities

arXiv 2506.06406 ↗First posted 6 Jun 2025cs.CL cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Mixture of Experts (MoE) architectures have become a key approach for scaling large language models, with growing interest in extending them to multimodal tasks. Existing methods to build multimodal MoE models either incur high training costs or suffer from degraded language capabilities when adapting pretrained models. To address this, we propose Soft ModalityAware Routing (SMAR), a novel regularization technique that uses Kullback Leibler divergence to control routing probability distributions across modalities, encouraging expert specialization without modifying model architecture or heavily relying on textual data. Experiments on visual instruction tuning show that SMAR preserves language ability at 86.6% retention with only 2.5% pure text, outperforming baselines while maintaining strong multimodal performance. Our approach offers a practical and efficient solution to balance modality differentiation and language capabilities in multimodal MoE models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 31 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

24 of 31 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  2. 2024Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts16 Nov
  3. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training10 Oct
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  5. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLM9 Aug
  6. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  9. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 15

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  10. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  11. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  12. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
3 more
  1. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  2. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 70
  3. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Mixture of Experts in Large Language Modelscited by 15
  2. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMscited by 2
  3. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MoExtend: Tuning New Experts for Modality and Task Extension
  2. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
  3. 2025Routing-Based Continual Learning for Multimodal Large Language Models
  4. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  5. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
  6. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models
  7. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  8. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping