mlsanityData to 14 Oct 2026

Enhancing Multi-modal Models with Heterogeneous MoE Adapters for Fine-tuning

arXiv 2503.20633 ↗First posted 26 Mar 2025cs.LGcited by 3

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Multi-modal models excel in cross-modal tasks but are computationally expensive due to their billions of parameters. Parameter-efficient fine-tuning (PEFT) offers a solution by adding small trainable components while freezing pre-trained parameters. However, existing methods primarily focus on uni-modal processing, overlooking the critical modal fusion needed for multi-modal tasks. To fill this gap, we propose heterogeneous mixture of experts adapters that extend the traditional PEFT framework to support multi-modal expert combinations and improve information interaction. Additionally, our approach modifies the affine linear expert design to enable efficient modal fusion in a low-rank space, achieving competitive performance with only 5-8% of the parameters fine-tuned. Experiments across eight downstream tasks, including visual-audio and text-visual, demonstrate the superior performance of the approach.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 37 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

27 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  2. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE23 Aug
  3. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun

Other references · 24

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  12. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
12 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  3. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  4. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 278
  5. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  6. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 191
  7. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  8. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
  9. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  10. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  11. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  12. 2024MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Modelscited by 4
  2. 2025AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Modelscited by 3
  3. 2026Hyperbolic and Evidence-Prioritized Experts for Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking
  2. 2026From Experts to Sub-experts: Fine-grained Parameter-Efficient Fine-Tuning for MoE LLMs
  3. 2026ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
  4. 2025MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models
  5. 2024Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models
  6. 2024MoExtend: Tuning New Experts for Modality and Task Extension
  7. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
  8. 2026SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs