mlsanityData to 7 Oct 2026

Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading

arXiv 2502.05370 ↗First posted 7 Feb 2025cs.LG cs.AI cs.DCcited by 24

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have gained immense success in revolutionizing various applications, including content generation, search and recommendation, and AI-assisted operation. To reduce high training costs, Mixture-of-Experts (MoE) architecture has become a popular backbone for modern LLMs. However, despite the benefits, serving MoE-based LLMs experience severe memory inefficiency due to sparsely activated experts. Recent studies propose to offload inactive experts from GPU memory to CPU memory to improve the serving efficiency of MoE models. However, they either incur high inference latency or high model memory footprints due to coarse-grained designs. To tame the latency-memory trade-off in MoE serving, we present FineMoE, a fine-grained expert offloading system for MoE serving that achieves low inference latency with memory efficiency. We design FineMoE to extract fine-grained expert selection patterns from MoE models and semantic hints from input prompts to efficiently guide expert prefetching, caching, and offloading decisions. FineMoE is prototyped on top of HuggingFace Transformers and deployed on a six-GPU testbed. Experiments with open-source MoE models and real-world workloads show that FineMoE reduces inference latency by 47% and improves expert hit rate by 39% over state-of-the-art solutions.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 41 of its 66 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

41 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Optimizing Distributed Deployment of Mixture-of-Experts Model Inference in Serverless Computing9 Jan
  2. 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
  3. 2024Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection13 Nov
  4. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  5. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  6. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design24 Oct
  7. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  8. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  9. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  10. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  11. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  12. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
17 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  5. 2023Using an LLM to Help With Code Understandingcited by 222
  6. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  7. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  8. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  9. 2024Data-efficient Fine-tuning for LLM-based Recommendationcited by 103
  10. 2024Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learningcited by 51
  11. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 37
  12. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  13. 2023LILAC: Log Parsing using LLMs with Adaptive Parsing Cachecited by 30
  14. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  15. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  16. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  17. 2024Go Static: Contextualized Logging Statement Generationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 19.

24 papers.

Sort:

By area:

  1. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  2. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  3. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decodingcited by 8
  4. 2025DuoServe-MoE: Dual-Phase Expert Prefetch and Caching for LLM Inference QoS Assurancecited by 4
  5. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  6. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  7. 2025Remoe: Towards Efficient and Low-Cost MoE Inference in Serverless Computingcited by 2
  8. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Modelscited by 2
  9. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inferencecited by 2
  10. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  11. 2026Speculating Experts Accelerates Inference for Mixture-of-Expertscited by 1
  12. 2026Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancycited by 1
12 more
  1. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 1
  2. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 1
  3. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Servingcited by 0
  4. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  5. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Modelscited by 0
  6. 2026TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimescited by 0
  7. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0
  8. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloadingcited by 0
  9. 2026BOOST: Concurrent Access to Host Memory and HBM to Accelerate LLM Inferencecited by 0
  10. 2026Physically Partitioned KVCache Format for CPU--GPU Load Balancing in MoE Inferencecited by 0
  11. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
  12. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Faster MoE LLM Inference for Extremely Large Models
  2. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  3. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference
  4. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
  5. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
  6. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  7. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Models
  8. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights