mlsanityData to 14 Oct 2026

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

arXiv 2609.38823 ↗First posted 30 Sep 2026cs.CVcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts also fits: Multimodal MoE models

Abstract (arXiv)

Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly because long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69x speedup. Code will be available at https://github.com/ShawnTan86/DecoMoE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 32 of its 35 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

32 of 35 references.

References · 32

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2025Qwen3-VL Technical Reportcited by 2,793
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  5. 2024Mixtral of Expertscited by 1,823
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  9. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  10. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  11. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  12. 2025Kimi-VL Technical Reportcited by 388
20 more
  1. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  3. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  4. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 166
  5. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 157
  6. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  7. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 139
  8. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 126
  9. 2024Multi-Stage Vision Token Dropping: Towards Efficient Multimodal Large Language Modelcited by 68
  10. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 62
  11. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  12. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  13. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  14. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 41
  15. 2025TokenCarve: Information-Preserving Visual Token Compression in Multimodal Large Language Modelscited by 19
  16. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findingscited by 15
  17. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skippingcited by 11
  18. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 9
  19. 2025FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruningcited by 2
  20. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
  2. 2026Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
  3. 2025AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
  4. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence
  5. 2026Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
  6. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts
  7. 2024ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts
  8. 2026MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs