# MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?

arXiv: https://arxiv.org/abs/2509.07727 · HTML: https://mlsanity.com/paper/2509.07727/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 9 Sep 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2402.14800/) (2024)
- [Fast Inference of Mixture-of-Experts Language Models with Offloading](https://mlsanity.com/paper/2312.17238/) (2023)
- [HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://mlsanity.com/paper/2411.01433/) (2024)
- [MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache](https://mlsanity.com/paper/2401.14361/) (2024)
- [A Survey on Inference Optimization Techniques for Mixture of Experts Models](https://mlsanity.com/paper/2412.14219/) (2024)
- [ProMoE: Fast MoE-based LLM Serving using Proactive Caching](https://mlsanity.com/paper/2410.22134/) (2024)
- [MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance](https://mlsanity.com/paper/2505.03804/) (2025)
- [MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design](https://mlsanity.com/paper/2505.05799/) (2025)
- [SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget](https://mlsanity.com/paper/2308.15030/) (2023)
- [ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling](https://mlsanity.com/paper/2410.17954/) (2024)
- [FedFa: A Fully Asynchronous Training Paradigm for Federated Learning](https://mlsanity.com/paper/2404.11015/) (2024)
- [MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators](https://mlsanity.com/paper/2504.02658/) (2025)
- [MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching](https://mlsanity.com/paper/2503.09716/) (2025)
- [An Optimized Error-controlled MPI Collective Framework Integrated with Lossy Compression](https://mlsanity.com/paper/2304.03890/) (2023)
- [HLoRA: Efficient Federated Learning System for LLM Heterogeneous Fine-Tuning](https://mlsanity.com/paper/2503.00813/) (2025)
- [ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression](https://mlsanity.com/paper/2502.18554/) (2025)
- [CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuning](https://mlsanity.com/paper/2502.18910/) (2025)

## Built on by (most cited first, 1 of 1)

- [SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning](https://mlsanity.com/paper/2604.26388/) (2026)

## Abstract (arXiv)

With the widespread application of Mixture of Experts (MoE) reasoning models in the field of LLM learning, efficiently serving MoE models under limited GPU memory constraints has emerged as a significant challenge. Offloading the non-activated experts to main memory has been identified as an efficient approach to address such a problem, while it brings the challenges of transferring the expert between the GPU memory and main memory. We need to explore an efficient approach to compress the expert and analyze how the compression error affects the inference performance. To bridge this gap, we propose employing error-bounded lossy compression algorithms (such as SZ3 and CuSZp) to compress non-activated experts, thereby reducing data transfer overhead during MoE inference. We conduct extensive experiments across various benchmarks and present a comprehensive analysis of how compression-induced errors in different experts affect overall inference accuracy. The results indicate that experts in the shallow layers, which are primarily responsible for the attention mechanism and the transformation of input tokens into vector representations, exhibit minimal degradation in inference accuracy when subjected to bounded errors. In contrast, errors in the middle-layer experts, which are central to model reasoning, significantly impair inference accuracy. Interestingly, introducing bounded errors in the deep-layer experts, which are mainly responsible for instruction following and output integration, can sometimes lead to improvements in inference accuracy.