mlsanityData to 14 Oct 2026

MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache

arXiv 2401.14361 ↗First posted 25 Jan 2024cs.LG cs.PFcited by 50

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

This paper presents MoE-Infinity, an efficient MoE inference system designed for personal machines with limited GPU memory capacity. The key idea for MoE-Infinity is that on personal machines, which are often single-user environments, MoE-based LLMs typically operate with a batch size of one. In this setting, MoE models exhibit a high degree of activation sparsity, meaning a small number of experts are frequently reused in generating tokens during the decode phase. Leveraging this idea, we design a sparsity-aware expert cache, which can trace the sparse activation of experts during inference and carefully select the trace that represents the sparsity pattern. By analyzing these selected traces, MoE-Infinity guides the replacement and prefetching of the expert cache, providing 3.1-16.7x per-token latency improvements over numerous state-of-the-art systems, including vLLM, Ollama, DeepSpeed and BrainStorm across various MoE models (DeepSeek and Mixtral) when handling different LLM tasks. MoE-Infinity's source code is publicly available at https://github.com/EfficientMoE/MoE-Infinity

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 6 of its 25 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

6 of 25 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec

Other references · 5

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 39.

50 papers.

Sort:

By area:

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  2. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  3. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inferencecited by 14
  4. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decodingcited by 12
  5. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  6. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9
  7. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  8. 2025Janus: Disaggregating Attention and Experts for Scalable MoE Inferencecited by 8
  9. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placementcited by 7
  10. 2025BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inferencecited by 5
  11. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 4
  12. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
38 more
  1. 2026Temporally Extended Mixture-of-Experts Modelscited by 3
  2. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  3. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inferencecited by 3
  4. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Expertscited by 3
  5. 2026FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Executioncited by 3
  6. 2026Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inferencecited by 2
  7. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Modelscited by 2
  8. 2026Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUscited by 2
  9. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patternscited by 2
  10. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardwarecited by 2
  11. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inferencecited by 2
  12. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  13. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1
  14. 2026Speculating Experts Accelerates Inference for Mixture-of-Expertscited by 1
  15. 2026Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Studycited by 1
  16. 2026Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Modelscited by 1
  17. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  18. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Servingcited by 0
  19. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  20. 2026Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devicescited by 0
  21. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Modelscited by 0
  22. 2026HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inferencecited by 0
  23. 2026Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimescited by 0
  24. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inferencecited by 0
  25. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0
  26. 2026SAEM: Stage-Aware Expert Management for Memory-Efficient MoE Inference in Chain-of-Thought Reasoningcited by 0
  27. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0
  28. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloadingcited by 0
  29. 2026Mixture-of-Experts Language Models Can Be Strong and Efficient Retrieverscited by 0
  30. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
  31. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PCcited by 0
  32. 2026TopoCompress: Topology Aware Token Compression Algorithm for Distributed Edge MoE Inferencecited by 0
  33. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloadingcited by 0
  34. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0
  35. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0
  36. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Stagingcited by 0
  37. 2026Democratizing MoE inference on commodity GPUs with CoMoEcited by 0
  38. 2026SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints
  2. 2025Faster MoE LLM Inference for Extremely Large Models
  3. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  4. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  5. 2025MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models
  6. 2025Accelerating MoE Model Inference with Expert Sharding
  7. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  8. 2025Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems