mlsanityData to 14 Oct 2026

Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models

arXiv 2505.16056 ↗First posted 21 May 2025cs.LG cs.AIcited by 8

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) enables efficient scaling of large language models (LLMs) with sparsely activated experts during inference. To effectively deploy large MoE models on memory-constrained devices, many systems introduce *expert offloading* that caches a subset of experts in fast memory, leaving others on slow memory to run on CPU or load on demand. While some research has exploited the locality of expert activations, where consecutive tokens activate similar experts, the degree of this **local routing consistency** varies across models and remains understudied. In this paper, we propose two metrics to measure local routing consistency of MoE models: (1) **Segment Routing Best Performance (SRP)**, which evaluates how well a fixed group of experts can cover the needs of a segment of tokens, and (2) **Segment Cache Best Hit Rate (SCH)**, which measures the hit rate of an expert cache utilizing a length of future information under a cache limit. We analyze 20 MoE LLMs with diverse sizes and architectures and use toy models to verify key factors related to local routing consistency. We find a strong trade-off between local routing consistency and *local* load balance, while showing that *global* load balance can coexist with local routing consistency. Meanwhile, settings like shared experts that decrease expert combination space can lead to low local routing consistency. We further reveal that domain-specialized experts contribute more to routing consistency than vocabulary-specialized ones, and that most models balance between cache effectiveness and efficiency with cache sizes approximately twice the active experts. These findings pave the way for memory-efficient MoE design and deployment without compromising inference speed. We publish the code for replicating experiments at https://github.com/ljcleo/moe-lrc .

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 52 of its 63 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

52 of 63 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Yuan 2.0-M32: Mixture of Experts with Attention Router28 May
  2. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition4 Feb
  3. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
  4. 2023Soft Merging of Experts with Adaptive Routing6 Jun
  5. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  6. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 45

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2024Mixtral of Expertscited by 1,823
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  11. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  12. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 200
33 more
  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  2. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 147
  3. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 146
  4. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  5. 2022Accelerating Distributed MoE Training and Inference with Linacited by 109
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 104
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  8. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  9. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  10. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  11. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  12. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  13. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 57
  14. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  15. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  16. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 50
  17. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 48
  18. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  19. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inferencecited by 42
  20. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 41
  21. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 40
  22. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  23. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  24. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37
  25. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  26. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 28
  27. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  28. 2024LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Trainingcited by 26
  29. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  30. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  31. 2024GRIN: GRadient-INformed MoEcited by 17
  32. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  33. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inferencecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  2. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Modelscited by 2
  3. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inferencecited by 2
  4. 2026Multi-level context Modeling for consistent expert selection in Mixture-of-Expertscited by 1
  5. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  6. 2026CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Expertscited by 0
  7. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  8. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  2. 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
  3. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
  4. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
  5. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  6. 2026A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
  7. 2026Slicing and Dicing: Configuring Optimal Mixtures of Experts
  8. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know