# ProMoE: Fast MoE-based LLM Serving using Proactive Caching

arXiv: https://arxiv.org/abs/2410.22134 · HTML: https://mlsanity.com/paper/2410.22134/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time](https://mlsanity.com/paper/2310.17157/) (2023)
- [InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management](https://mlsanity.com/paper/2406.19707/) (2024)
- [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](https://mlsanity.com/paper/2312.12456/) (2023)
- [The Case for Learned Index Structures](https://mlsanity.com/paper/1712.01208/) (2017)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference](https://mlsanity.com/paper/2401.08671/) (2024)
- [Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference](https://mlsanity.com/paper/2308.12066/) (2023)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)
- [SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget](https://mlsanity.com/paper/2308.15030/) (2023)

## Built on by (most cited first, 30 of 40)

- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading](https://mlsanity.com/paper/2502.05370/) (2025)
- [HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference](https://mlsanity.com/paper/2504.05897/) (2025)
- [Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate](https://mlsanity.com/paper/2502.12224/) (2025)
- [SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference](https://mlsanity.com/paper/2510.10302/) (2025)
- [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](https://mlsanity.com/paper/2511.14102/) (2025)
- [FloE: On-the-Fly MoE Inference on Memory-constrained GPU](https://mlsanity.com/paper/2505.05950/) (2025)
- [Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism](https://mlsanity.com/paper/2509.08342/) (2025)
- [Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models](https://mlsanity.com/paper/2505.16056/) (2025)
- [LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers](https://mlsanity.com/paper/2509.23638/) (2025)
- [Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference](https://mlsanity.com/paper/2511.15015/) (2025)
- [MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching](https://mlsanity.com/paper/2503.09716/) (2025)
- [ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference](https://mlsanity.com/paper/2510.26730/) (2025)
- [Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts](https://mlsanity.com/paper/2505.21079/) (2025)
- [ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling](https://mlsanity.com/paper/2601.21198/) (2026)
- [SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution](https://mlsanity.com/paper/2508.18983/) (2025)
- [SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference](https://mlsanity.com/paper/2512.12990/) (2025)
- [Bandwidth-Efficient Adaptive Mixture-of-Experts via Low-Rank Compensation](https://mlsanity.com/paper/2512.17073/) (2025)
- [Temporally Extended Mixture-of-Experts Models](https://mlsanity.com/paper/2604.20156/) (2026)
- [FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution](https://mlsanity.com/paper/2608.16157/) (2026)
- [Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference](https://mlsanity.com/paper/2609.04895/) (2026)
- [CoMoE: Collaborative Optimization of Expert Aggregation and Offloading for MoE-based LLMs at Edge](https://mlsanity.com/paper/2508.09208/) (2025)
- [Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computing](https://mlsanity.com/paper/2512.18674/) (2025)
- [DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs](https://mlsanity.com/paper/2602.03495/) (2026)
- [CRAFT: Fine-Grained Cost-Aware Expert Replication For Efficient Mixture-of-Experts Serving](https://mlsanity.com/paper/2603.28768/) (2026)
- [Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU-GPU Hybrid Design](https://mlsanity.com/paper/2606.10493/) (2026)
- [MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?](https://mlsanity.com/paper/2509.07727/) (2025)
- [MoEless: Efficient MoE LLM Serving via Serverless Computing](https://mlsanity.com/paper/2603.06350/) (2026)
- [Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference](https://mlsanity.com/paper/2606.29982/) (2026)
- [Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling](https://mlsanity.com/paper/2603.27624/) (2026)

## Abstract (arXiv)

The promising applications of large language models are often limited by the constrained GPU memory capacity available on edge devices. Mixture-of-Experts (MoE) models help address this issue by activating only a subset of the model's parameters during computation. This approach allows the unused parameters to be offloaded to host memory, thereby reducing the overall GPU memory demand. However, existing cache-based offloading solutions handle cache misses reactively, which significantly impacts system performance. In this paper, we introduce ProMoE, a novel proactive caching system that utilizes intermediate results to predict subsequent expert usage. By proactively fetching experts in advance, ProMoE eliminates passive cache misses, removes loading time from the critical path, and reduces the performance overhead associated with offloading. Our evaluations demonstrate that ProMoE achieves an average speedup of 2.20x (up to 3.21x) and 2.07x (up to 5.02x) in the prefill and decode stages, respectively, compared to existing offloading solutions.