mlsanityData to 7 Oct 2026

ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling

arXiv 2410.17954 ↗First posted 23 Oct 2024cs.AI cs.CLcited by 18

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) models can outperform dense large language models at similar computation by activating only a small set of experts per token. However, stacking many expert modules introduces substantial parameter memory, which makes MoE models difficult to deploy in memory-constrained environments such as single-GPU devices. Offloading alleviates this issue by storing inactive experts in CPU memory and loading them on demand, but existing methods remain limited: static caches disregard input-dependent routing, and methods that train separate models to predict expert usage ahead of time are often inaccurate or require significant training cost. We propose ExpertFlow, a lightweight MoE inference system that addresses this routing dependency through three coordinated components: 1) a transformer-based routing path predictor that estimates expert usage across all MoE layers in a single forward pass, 2) a token scheduler that groups tokens with similar predicted routes to improve expert utilization, and 3) a predictive expert cache that loads only the required experts while correcting mispredictions at runtime. Together, these components enable efficient expert loading and execution, reducing GPU memory usage by up to 93.72% and improving inference throughput by up to 10x over strong offloading baselines on a single GPU.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 37 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

22 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  4. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct
  5. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May

Other references · 17

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  12. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
5 more
  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  2. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 69
  3. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  4. 2024Cached Model-as-a-Resource: Provisioning Large Language Model Agents for Edge Intelligence in Space-air-ground Integrated Networkscited by 15
  5. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 14.

18 papers.

Sort:

By area:

  1. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  2. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 3
  3. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  4. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  5. 2026Routing-Free Mixture-of-Expertscited by 2
  6. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  7. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 1
  8. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 1
  9. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  10. 2026OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scalecited by 0
  11. 2026Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Expertscited by 0
  12. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
6 more
  1. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Modelscited by 0
  2. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrieverscited by 0
  3. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloadingcited by 0
  4. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0
  5. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
  6. 2026QCATS: Query Context-Aware Transformer Slicing for Efficient Predictive Query Processingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  2. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline
  4. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
  5. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  6. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  7. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  8. 2025Accelerating MoE Model Inference with Expert Sharding