mlsanityData to 14 Oct 2026

ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling

arXiv 2601.21198 ↗First posted 29 Jan 2026cs.DC cs.AI cs.LGcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

While Mixture-of-Experts (MoE) architectures substantially bolster the expressive power of large-language models, their prohibitive memory footprint severely impedes the practical deployment on resource-constrained edge devices, especially when model behavior must be preserved without relying on lossy quantization. In this paper, we present ZipMoE, an efficient and semantically lossless on-device MoE serving system. ZipMoE exploits the synergy between the hardware properties of edge devices and the statistical redundancy inherent to MoE parameters via a caching-scheduling co-design with provable performance guarantee. Fundamentally, our design shifts the paradigm of on-device MoE inference from an I/O-bound bottleneck to a compute-centric workflow that enables efficient parallelization. We implement a prototype of ZipMoE and conduct extensive experiments on representative edge computing platforms using popular open-source MoE models and real-world workloads. Our evaluation reveals that ZipMoE achieves up to 72.77% inference latency reduction and up to 6.76× higher throughput than the state-of-the-art systems.Our code is available at: https://github.com/npnothard/ZipMoE-ICML26.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 33 of its 55 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

33 of 55 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPU9 May
  2. 2025D2MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving17 Apr
  3. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  4. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  6. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  7. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  8. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  9. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  10. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  11. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 22

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  5. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  6. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  8. 2023On the Exploitability of Instruction Tuningcited by 116
  9. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 73
  10. 2024Exploiting LLM Quantizationcited by 65
  11. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  12. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
10 more
  1. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 30
  2. 2023SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budgetcited by 27
  3. 2024Quantization without Tearscited by 25
  4. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 24
  5. 2024Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomescited by 14
  6. 2024NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networkscited by 14
  7. 2024ZipNN: Lossless Compression for AI Modelscited by 14
  8. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inferencecited by 9
  9. 2020Scheduling with Communication Delays via LP Hierarchies and Clusteringcited by 2
  10. 2020Scheduling Precedence-Constrained Jobs on Related Machines with Communication Delaycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Lossless Tensor Compression as Program Synthesiscited by 1
  2. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Servingcited by 0
  3. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  4. 2026EStream: Fast and Memory-Efficient MoE Prefill through Expert Virtualization on Mobile NPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
  2. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
  3. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  4. 2025SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference
  5. 2026The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
  6. 2025Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
  7. 2026Beyond Uniform Experts: Cost-Aware Expert Execution for Efficient Multi-Device MoE Inference
  8. 2026MobileMoE: Scaling On-Device Mixture of Experts