mlsanityData to 14 Oct 2026

MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators

arXiv 2504.02658 ↗First posted 3 Apr 2025cs.LGcited by 8

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

A critical approach for efficiently deploying Mixture-of-Experts (MoE) models with massive parameters is quantization. However, state-of-the-art MoE models suffer from non-negligible accuracy loss with extreme quantization, such as under 4 bits. To address this, we introduce MiLo, a novel method that augments highly quantized MoEs with a mixture of low-rank compensators. These compensators consume only a small amount of additional memory but significantly recover accuracy loss from extreme quantization. MiLo also identifies that MoEmodels exhibit distinctive characteristics across weights due to their hybrid dense-sparse architectures, and employs adaptive rank selection policies along with iterative optimizations to close the accuracy gap. MiLo does not rely on calibration data, allowing it to generalize to different MoE models and datasets without overfitting to a calibration set. To avoid the hardware inefficiencies of extreme quantization, such as 3-bit, MiLo develops Tensor Core-friendly 3-bit kernels, enabling measured latency speedups on 3-bit quantized MoE models. Our evaluation shows that MiLo outperforms existing methods on SoTA MoE models across various tasks.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 43 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

26 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2024Mixtral of Expertscited by 1,823
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
12 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  5. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  6. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  7. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  8. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  9. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  10. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  11. 2023Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustnesscited by 34
  12. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 9
  2. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 2
  3. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?cited by 1
  4. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 1
  5. 2026BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantizationcited by 0
  6. 2026Every Expert Counts: ExactMoE for Memory-Efficient W4A16 Inferencecited by 0
  7. 2026ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Modelscited by 0
  8. 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
  2. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
  3. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
  4. 2024QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
  5. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  6. 2026MonoMoE: An Efficient Fused Mega-kernel for Quantized MoE Decoding
  7. 2025LExI: Layer-Adaptive Active Experts for Efficient MoE Model Inference
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference