mlsanityData to 14 Oct 2026

PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compression

arXiv 2610.13982 ↗First posted 23 Sep 2026cs.LG stat.MLcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) models have emerged as an effective paradigm for scaling large language models, yet their massive parameter footprint poses severe memory challenges for deployment. Existing compression methods typically treat expert pruning and merging as disjoint operations and rely on greedy heuristics, often leading to irreversible information loss or representational pollution. In this paper, we propose PRIME, a unified optimization framework for MoE compression via adaptive PRunIng and MErging. PRIME casts the discrete prune-merge assignment step as a global Integer Linear Programming over a frequency-weighted surrogate objective, jointly selecting pruning and merging decisions while accounting for expert usage. We motivate the frequency-aware objective through an error-bound analysis under fixed representative assignments, and show that the formulation favors high-frequency experts as centroids, promoting routing stability. Extensive experiments on Qwen3-30B-A3B and Qwen3.5-122B-A10B demonstrate that PRIME achieves strong accuracy retention across knowledge, coding, math, and long-context tasks, while serving benchmarks of merged models quantify the memory and throughput benefits of expert compression in vLLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 32 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

27 of 32 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025MergeMoE: Efficient Compression of MoE Models via Expert Output Merging16 Oct
  2. 2025REAP the Experts: Why Pruning Prevails for One-Shot MoE compression15 Oct
  3. 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning19 Sep
  4. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models10 Apr
  5. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  6. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering11 Oct
  7. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More8 Oct
  8. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts12 Jul
  9. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs1 Jul
  10. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb

Other references · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
5 more
  1. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  2. 2017Exploring Generalization in Deep Learningcited by 600
  3. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  4. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  5. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  2. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
  3. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Models
  4. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition
  5. 2026Does a Global Perspective Help Prune Sparse MoEs Elegantly?
  6. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
  7. 2026Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression
  8. 2026Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression