# MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache

arXiv: https://arxiv.org/abs/2401.14361 · HTML: https://mlsanity.com/paper/2401.14361/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 6 of its 25 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)

## Builds on (most cited first, 6 of 6)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)

## Built on by (most cited first, 30 of 50)

- [HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://mlsanity.com/paper/2411.01433/) (2024)
- [Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading](https://mlsanity.com/paper/2502.05370/) (2025)
- [Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference](https://mlsanity.com/paper/2412.00099/) (2024)
- [MoE-Spec: Expert Budgeting for Efficient Speculative Decoding](https://mlsanity.com/paper/2602.16052/) (2026)
- [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](https://mlsanity.com/paper/2511.14102/) (2025)
- [FloE: On-the-Fly MoE Inference on Memory-constrained GPU](https://mlsanity.com/paper/2505.05950/) (2025)
- [Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models](https://mlsanity.com/paper/2505.16056/) (2025)
- [Janus: Disaggregating Attention and Experts for Scalable MoE Inference](https://mlsanity.com/paper/2512.13525/) (2025)
- [Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement](https://mlsanity.com/paper/2508.12851/) (2025)
- [BuddyMoE: Exploiting Expert Redundancy to Accelerate Memory-Constrained Mixture-of-Experts Inference](https://mlsanity.com/paper/2511.10054/) (2025)
- [Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2511.10676/) (2025)
- [ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling](https://mlsanity.com/paper/2601.21198/) (2026)
- [Temporally Extended Mixture-of-Experts Models](https://mlsanity.com/paper/2604.20156/) (2026)
- [ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference](https://mlsanity.com/paper/2605.27081/) (2026)
- [A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference](https://mlsanity.com/paper/2606.15453/) (2026)
- [Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2607.12696/) (2026)
- [FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution](https://mlsanity.com/paper/2608.16157/) (2026)
- [Harvest: Opportunistic Peer-to-Peer GPU Caching for LLM Inference](https://mlsanity.com/paper/2602.00328/) (2026)
- [MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models](https://mlsanity.com/paper/2602.11192/) (2026)
- [Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs](https://mlsanity.com/paper/2604.18788/) (2026)
- [Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns](https://mlsanity.com/paper/2604.23150/) (2026)
- [WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware](https://mlsanity.com/paper/2606.21868/) (2026)
- [Sticky Routing: Training MoE Models for Memory-Efficient Inference](https://mlsanity.com/paper/2607.08780/) (2026)
- [MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?](https://mlsanity.com/paper/2509.07727/) (2025)
- [MoEless: Efficient MoE LLM Serving via Serverless Computing](https://mlsanity.com/paper/2603.06350/) (2026)
- [Speculating Experts Accelerates Inference for Mixture-of-Experts](https://mlsanity.com/paper/2603.19289/) (2026)
- [Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study](https://mlsanity.com/paper/2606.21428/) (2026)
- [Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models](https://mlsanity.com/paper/2608.04454/) (2026)
- [Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study](https://mlsanity.com/paper/2608.18261/) (2026)
- [FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving](https://mlsanity.com/paper/2604.02715/) (2026)

## Abstract (arXiv)

This paper presents MoE-Infinity, an efficient MoE inference system designed for personal machines with limited GPU memory capacity. The key idea for MoE-Infinity is that on personal machines, which are often single-user environments, MoE-based LLMs typically operate with a batch size of one. In this setting, MoE models exhibit a high degree of activation sparsity, meaning a small number of experts are frequently reused in generating tokens during the decode phase. Leveraging this idea, we design a sparsity-aware expert cache, which can trace the sparse activation of experts during inference and carefully select the trace that represents the sparsity pattern. By analyzing these selected traces, MoE-Infinity guides the replacement and prefetching of the expert cache, providing 3.1-16.7x per-token latency improvements over numerous state-of-the-art systems, including vLLM, Ollama, DeepSpeed and BrainStorm across various MoE models (DeepSeek and Mixtral) when handling different LLM tasks. MoE-Infinity's source code is publicly available at https://github.com/EfficientMoE/MoE-Infinity