mlsanityData to 7 Oct 2026

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

arXiv 2402.07033 ↗First posted 10 Feb 2024cs.LG cs.AI cs.DCcited by 69

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Large Language Models (LLMs) with the Mixture-of-Experts (MoE) architectures have shown promising performance on various tasks. However, due to the huge model sizes, running them in resource-constrained environments where the GPU memory is not abundant is challenging. Some existing systems propose to use CPU resources to solve that, but they either suffer from the significant overhead of frequently moving data between CPU and GPU, or fail to consider distinct characteristics of CPUs and GPUs. This paper proposes Fiddler, a resource-efficient inference system for MoE models with limited GPU resources. Fiddler strategically utilizes CPU and GPU resources by determining the optimal execution strategy. Our evaluation shows that, unlike state-of-the-art systems that optimize for specific scenarios such as single batch inference or long prefill, Fiddler performs better in all scenarios. Compared against different baselines, Fiddler achieves 1.26 times speed up in single batch inference, 1.30 times in long prefill processing, and 11.57 times in beam search inference. The code of Fiddler is publicly available at https://github.com/efeslab/fiddler.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 22 of its 43 references that are in the map. First posted on arXiv 10 Feb 2024.

Builds on

22 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  8. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
8 more
  1. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  2. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  4. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  5. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  6. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  7. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  8. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 25; 2026: 33.

69 papers.

Sort:

By area:

  1. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  2. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  3. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  4. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  5. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  6. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  7. 2024BlendServe: Optimizing Offline Inference for Auto-regressive Large Models with Resource-aware Batchingcited by 19
  8. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Schedulingcited by 18
  9. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Servingcited by 17
  10. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selectioncited by 16
  11. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  12. 2025Seesaw: High-throughput LLM Inference via Model Re-shardingcited by 15
38 more
  1. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  2. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inferencecited by 14
  3. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systemscited by 11
  4. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  5. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  6. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  7. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  8. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  9. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  10. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  11. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  12. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5
  13. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  14. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  15. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  16. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraintscited by 3
  17. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  18. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  19. 2025DiSCo: Device-Server Collaborative LLM-Based Text Streaming Servicescited by 2
  20. 2025Context-Aware Mixture-of-Experts Inference on CXL-Enabled GPU-NDP Systemscited by 2
  21. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  22. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  23. 2026TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offloadcited by 2
  24. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inferencecited by 2
  25. 2025Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impactscited by 1
  26. 2025Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systemscited by 1
  27. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systemscited by 1
  28. 2025Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelismcited by 1
  29. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devicescited by 1
  30. 2026MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Servingcited by 1
  31. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 1
  32. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  33. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inferencecited by 1
  34. 2026Noesis: Bidirectional Graph-RAG with Adaptive Parallelism and Cross-Knowledge-Base Semantic Discoverycited by 1
  35. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 1
  36. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  37. 2025QoS-Efficient Serving of Multiple Mixture-of-Expert LLMs Using Partial Runtime Reconfigurationcited by 0
  38. 2025Efficient Edge LLMs Deployment via HessianAware Quantization and CPU GPU Collaborativecited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  2. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  3. 2025Accelerating MoE Model Inference with Expert Sharding
  4. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  5. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  6. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  7. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  8. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference