mlsanityData to 7 Oct 2026

ProMoE: Fast MoE-based LLM Serving using Proactive Caching

arXiv 2410.22134 ↗First posted 29 Oct 2024cs.DC cs.AIcited by 38

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

The promising applications of large language models are often limited by the constrained GPU memory capacity available on edge devices. Mixture-of-Experts (MoE) models help address this issue by activating only a subset of the model's parameters during computation. This approach allows the unused parameters to be offloaded to host memory, thereby reducing the overall GPU memory demand. However, existing cache-based offloading solutions handle cache misses reactively, which significantly impacts system performance. In this paper, we introduce ProMoE, a novel proactive caching system that utilizes intermediate results to predict subsequent expert usage. By proactively fetching experts in advance, ProMoE eliminates passive cache misses, removes loading time from the critical path, and reduces the performance overhead associated with offloading. Our evaluations demonstrate that ProMoE achieves an average speedup of 2.20x (up to 3.21x) and 2.07x (up to 5.02x) in the prefill and decode stages, respectively, compared to existing offloading solutions.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 24 of its 42 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

24 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  2. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
10 more
  1. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  6. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  7. 2017The Case for Learned Index Structurescited by 188
  8. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 18.

38 papers.

Sort:

By area:

  1. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
  2. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  3. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inferencecited by 16
  4. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  5. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inferencecited by 12
  6. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  7. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  8. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanismcited by 9
  9. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  10. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Serverscited by 8
  11. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batchingcited by 6
  12. 2025Dynamic Expert Quantization for Scalable Mixture-of-Experts Inferencecited by 6
26 more
  1. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inferencecited by 5
  2. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Expertscited by 4
  3. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  4. 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitutioncited by 3
  5. 2025Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensationcited by 3
  6. 2025CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edgecited by 2
  7. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  8. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCscited by 2
  9. 2026CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Servingcited by 2
  10. 2026Temporally Extended Mixture-of-Experts Modelscited by 2
  11. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inferencecited by 2
  12. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  13. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inferencecited by 1
  14. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  15. 2026Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Designcited by 1
  16. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inferencecited by 1
  17. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 1
  18. 2026Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Schedulingcited by 0
  19. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  20. 2026SmartMage: Dynamic Modality Orchestration for 3D Scene Understandingcited by 0
  21. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inferencecited by 0
  22. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0
  23. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloadingcited by 0
  24. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PCcited by 0
  25. 2026A Multi-Engine Dataflow for MoE Decoding on Scratchpad-Based Tensor Acceleratorscited by 0
  26. 2026Cost-Aware Mixture-of-Experts Coordination for Model Marketscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
  2. 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading
  3. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
  4. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  5. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  6. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
  7. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
  8. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints