mlsanityData to 14 Oct 2026

EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models

arXiv 2508.01625 ↗First posted 3 Aug 2025cs.LG cs.AIcited by 18

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs. However, it is hindered by two critical challenges: (1) substantial GPU memory consumption to load all experts; (2) low activated parameters cannot be equivalently translated into inference acceleration effects. In this work, we propose EAC-MoE, an Expert-Selection Aware Compressor for MoE-LLMs, which deeply aligns with the characteristics of MoE from the perspectives of quantization and pruning, and introduces two modules to address these two challenges respectively: (1) The expert selection bias caused by low-bit quantization is a major factor contributing to the performance degradation in MoE-LLMs. Based on this, we propose Quantization with Expert-Selection Calibration (QESC), which mitigates the expert selection bias by calibrating the routers within the MoE; (2) There are always certain experts that are not crucial for the corresponding tasks, yet causing inference latency. Therefore, we propose Pruning based on Expert-Selection Frequency (PESF), which significantly improves inference speed by pruning less frequently used experts for current task. Extensive experiments demonstrate that our approach significantly reduces memory usage and improves inference speed with minimal performance degradation.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 48 of its 64 references that are in the map. First posted on arXiv 3 Aug 2025.

Builds on

48 of 64 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  2. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More8 Oct
  3. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  4. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  5. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  6. 2024Scaling Laws for Fine-Grained Mixture of Experts12 Feb
  7. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  8. 2021Efficient Large Scale Language Modeling with Mixtures of Experts20 Dec
  9. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models22 Sep
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
26 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2024Qwen2 Technical Reportcited by 2,120
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  11. 2021Measuring Coding Challenge Competence With APPScited by 902
  12. 2020Aligning AI With Shared Human Valuescited by 723
  13. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  14. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  15. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  16. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  17. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  18. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  19. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  20. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  21. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  22. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  23. 2024BiLLM: Pushing the Limit of Post-Training Quantization for LLMscited by 144
  24. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 97
  25. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 64
  26. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Expertscited by 40

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 16.

18 papers.

Sort:

By area:

  1. 2025DartQuant: Efficient Rotational Distribution Calibration for LLM Quantizationcited by 17
  2. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Modelscited by 8
  3. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMscited by 7
  4. 2026AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantizationcited by 7
  5. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  6. 2026Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Modelscited by 6
  7. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 5
  8. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  9. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  10. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  11. 2026MoEITS: A Green AI approach for simplifying MoE-LLMscited by 1
  12. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tilingcited by 1
6 more
  1. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routingcited by 1
  2. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compressioncited by 0
  3. 2026Pricing the Risk of Runtime Compression: Anytime-Valid Admission and a Served-Output Law for Compressed Serving Statecited by 0
  4. 2026HDA-MoE: Hybrid Parallelism and Dynamic, Adaptive Scheduling for Mixture-of-Experts with 3D Near-Memory Processingcited by 0
  5. 2026SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PCcited by 0
  6. 2026You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MC#: Mixture Compressor for Mixture-of-Experts Large Models
  2. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  4. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  5. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  6. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
  7. 2026PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression
  8. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models