mlsanityData to 14 Oct 2026

HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference

arXiv 2504.05897 ↗First posted 8 Apr 2025cs.LG cs.DCcited by 16

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

The Mixture of Experts (MoE) architecture has demonstrated significant advantages as it enables to increase the model capacity without a proportional increase in computation. However, the large MoE model size still introduces substantial memory demands, which usually requires expert offloading on resource-constrained platforms and incurs significant overhead. Hybrid CPU-GPU inference has been proposed to leverage CPU computation to reduce expert loading overhead but faces major challenges: on one hand, the expert activation patterns of MoE models are highly unstable, rendering the fixed mapping strategies in existing works inefficient; on the other hand, the hybrid CPU-GPU schedule for MoE is inherently complex due to the diverse expert sizes, structures, uneven workload distribution, etc. To address these challenges, in this paper, we propose HybriMoE, a hybrid CPU-GPU inference framework that improves resource utilization through a novel CPU-GPU scheduling and cache management system. HybriMoE introduces (i) a dynamic intra-layer scheduling strategy to balance workloads across CPU and GPU, (ii) an impact-driven inter-layer prefetching algorithm, and (iii) a score-based caching algorithm to mitigate expert activation instability. We implement HybriMoE on top of the kTransformers framework and evaluate it on three widely used MoE-based LLMs. Experimental results demonstrate that HybriMoE achieves an average speedup of 1.33× in the prefill stage and 1.70× in the decode stage compared to state-of-the-art hybrid MoE inference framework. Our code is available at: https://github.com/PKU-SEC-Lab/HybriMoE.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 19 of its 30 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

19 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  2. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  3. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  4. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  5. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024Mixtral of Expertscited by 1,823
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  12. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 22
1 more
  1. 2023Adaptive Gating in Mixture-of-Experts based Language Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  2. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Accelerationcited by 8
  3. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  4. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inferencecited by 4
  5. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  6. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 3
  7. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routingcited by 2
  8. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  9. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  10. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 2
  11. 2026Noesis: Bidirectional Graph-RAG with Adaptive Parallelism and Cross-Knowledge-Base Semantic Discoverycited by 1
  12. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
4 more
  1. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devicescited by 0
  2. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processingcited by 0
  3. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0
  4. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  2. 2025Static Batching of Irregular Workloads on GPUs: Framework and Application to Efficient MoE Model Inference
  3. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  4. 2026TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
  5. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
  6. 2026GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
  7. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  8. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts