mlsanityData to 14 Oct 2026

Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models

arXiv 2608.04454 ↗First posted 5 Aug 2026cs.CV cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Mixture-of-experts vision-language models (MoE-VLMs) increase model capacity with sparse expert activation, yet deployment requires storing the full expert pool. Training-free expert merging reduces this burden, and many routing-based methods aggregate routing statistics across all tokens to determine merge compatibility. However, MoE-VLM inference is phase-structured: image-context tokens carry visual content, question tokens specify the query, and answer tokens produce the output, with different counts and routing distributions. Because image-context tokens are far more numerous, global aggregation can overemphasize image-context processing and obscure phase-conditioned expert roles, making experts serving different phases appear interchangeable and degrading model performance. We therefore argue that MoE-VLM expert merging should preserve phase-conditioned expert roles, judging compatibility by how experts serve different phases rather than globally aggregated routing statistics. Based on this view, we propose RoleMerge, a training-free method that constructs each expert's Routing Role Profile (RRP) from phase-normalized routing statistics, capturing its relative phase preference. Guided by expert-phase information loss, RoleMerge merges experts with compatible profiles and their corresponding router entries while preserving answer-decoding expert distinctions. Experiments on three models and multiple benchmarks show that RoleMerge preserves more of the full model's performance than alternative expert-merging methods at matched expert-retention ratios, with relative improvements of up to 9.6 percent in six-task macro-average performance. These results validate phase-conditioned expert roles as a more effective basis than global routing aggregation for MoE-VLM expert merging.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 35 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec

Other references · 23

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  10. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  12. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
11 more
  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  2. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  4. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 50
  5. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compressioncited by 44
  6. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clusteringcited by 35
  7. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
  8. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Mergingcited by 10
  9. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  10. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 9
  11. 2025CAMEx: Curvature-aware Merging of Expertscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
  2. 2026Self-Routing: Parameter-Free Expert Routing from Hidden States
  3. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
  4. 2024Routers in Vision Mixture of Experts: An Empirical Study
  5. 2025Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
  6. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model
  7. 2025How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
  8. 2026L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts