mlsanityData to 7 Oct 2026

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

arXiv 2605.05899 ↗First posted 7 May 2026cs.LGcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

also in 1 more
Abstract (arXiv)

Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as textitvisual-expert affinity: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 53 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

36 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning22 Nov
  2. 2025MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping19 Nov
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan

Other references · 32

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  9. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  11. 2023Multimodal Large Language Models: A Surveycited by 193
  12. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
20 more
  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 88
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  4. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  5. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 48
  6. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 38
  7. 2025Similarity-Aware Token Pruning: Your VLM but Fastercited by 32
  8. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  9. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  10. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  11. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  12. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  13. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  14. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  15. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  16. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  17. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 4
  18. 2026SpecMD: A Comprehensive Study On Speculative Expert Prefetchingcited by 3
  19. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systemscited by 2
  20. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Expertscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert
  2. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  3. 2026DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
  4. 2026VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
  5. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  6. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  7. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
  8. 2026MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference