mlsanityData to 7 Oct 2026

Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models

arXiv 2407.14093 ↗First posted 19 Jul 2024cs.MMcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Recently, mixture of experts (MoE) has become a popular paradigm for achieving the trade-off between modal capacity and efficiency of multi-modal large language models (MLLMs). Different from previous efforts, we are dedicated to exploring the dynamic expert path in an already exist MLLM and show that a standard MLLM can be also a mixture of experts. To approach this target, we propose a novel dynamic expert scheme for MLLMs, termed Routing Experts (RoE), which can achieve example-dependent optimal path routing without obvious structure tweaks. Meanwhile, a new regularization of structure sparsity is also introduced to enforce MLLMs to learn more short-cut inference, ensuring the efficiency. In addition, we also realize the first attempt of aligning the training and inference schemes of MLLMs in terms of network routing. To validate RoE, we apply it to a set of latest MLLMs, including LLaVA-1.5, LLaVA-HR and VILA, and conduct extensive experiments on a bunch of VL benchmarks. The experiment results not only show the great advantages of our RoE in improving MLLMs' efficiency, but also yield obvious advantages than MoE-LLaVA in both performance and speed, e.g., an average performance gain of 3.3% on 5 benchmarks while being faster.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 73 references that are in the map. First posted on arXiv 19 Jul 2024.

Builds on

63 of 73 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  2. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models5 Mar
  3. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning19 Dec
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
5 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  3. 2023Generating Images with Multimodal Language Models26 May
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Scaling Vision-Language Models with Sparse Mixture of Experts13 Mar

Other references · 46

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
34 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  6. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  7. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  9. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  10. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  11. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  13. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  14. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  15. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  16. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  17. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  18. 2021Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inferencecited by 96
  19. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  20. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 68
  21. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  22. 2023Parameter-efficient Tuning of Large-scale Multimodal Foundation Modelcited by 23
  23. 2023DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasetscited by 23
  24. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  25. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  26. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  27. 2022RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrievalcited by 11
  28. 2023SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inferencecited by 9
  29. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  30. 2023CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devicescited by 8
  31. 2023CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrievalcited by 7
  32. 2022A Survivor in the Era of Large-Scale Pretraining: An Empirical Study of One-Stage Referring Expression Comprehensioncited by 5
  33. 2023Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Modelscited by 5
  34. 2024Radial Networks: Dynamic Layer Routing for High-Performance Large Language Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 5.

13 papers.

Sort:

By area:

  1. 2025Advancing Expert Specialization for Better MoEcited by 37
  2. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  3. 2025Life-Cycle Routing Vulnerabilities of LLM Routercited by 10
  4. 2025Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layerscited by 7
  5. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6
  6. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  7. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Modelscited by 2
  8. 2026CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answeringcited by 1
  9. 2025Dynamic Pyramid Network for Efficient Multimodal Large Language Modelcited by 0
  10. 2025Knowledge Grafting: A Mechanism for Optimizing AI Model Deployment in Resource-Constrained Environmentscited by 0
  11. 2025InstructMoLE: Instruction-Guided Mixture of Low-rank Experts for Multi-Conditional Image Generationcited by 0
  12. 2026FedMPT: Federated Multi-label Prompt Tuning of Vision-Language Modelscited by 0
1 more
  1. 2026CogniRoute: Learning to Route Social Evidence in Omni-Modal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multilingual Routing in Mixture-of-Experts
  2. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  3. 2022StableMoE: Stable Routing Strategy for Mixture of Experts
  4. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  5. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  6. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
  7. 2026Mixture of Experts for Low-Resource LLMs
  8. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models