# ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression

arXiv: https://arxiv.org/abs/2605.29350 · HTML: https://mlsanity.com/paper/2605.29350/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 23 of its 27 references that are in the map.
- First posted on arXiv 28 May 2026.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 23 of 23)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale](https://mlsanity.com/paper/2201.05596/) (2022)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2402.14800/) (2024)
- [Task-Specific Expert Pruning for Sparse Mixture-of-Experts](https://mlsanity.com/paper/2206.00277/) (2022)
- [MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition](https://mlsanity.com/paper/2411.01016/) (2024)
- [REAP the Experts: Why Pruning Prevails for One-Shot MoE compression](https://mlsanity.com/paper/2510.13999/) (2025)
- [Retraining-Free Merging of Sparse MoE via Hierarchical Clustering](https://mlsanity.com/paper/2410.08589/) (2024)
- [DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning](https://mlsanity.com/paper/2509.16105/) (2025)
- [MergeMoE: Efficient Compression of MoE Models via Expert Output Merging](https://mlsanity.com/paper/2510.14436/) (2025)
- [EvoESAP: Non-Uniform Expert Pruning for Sparse MoE](https://mlsanity.com/paper/2603.06003/) (2026)
- [UniPool: A Globally Shared Expert Pool for Mixture-of-Experts](https://mlsanity.com/paper/2605.06665/) (2026)

## Built on by (most cited first, 1 of 1)

- [RAZOR: Pruning Replaceable Experts in LLMs](https://mlsanity.com/paper/2609.30465/) (2026)

## Abstract (arXiv)

Sparse Mixture-of-Experts (MoE) models scale model capacity through sparse activation, but storing their full parameters remains costly in GPU memory. Existing expert compression methods address this problem through pruning, merging, or substitution, typically coupling expert pool construction with the assignment of routed workloads. In this work, we formulate expert compression as two decisions--constructing an expert pool and assigning the original experts' workloads to that pool--and show that assignment can be optimized after the pool is fixed. Based on this formulation, we introduce ConMoE, which constructs a compact pool using Anchor Coverage and optimizes assignment through a router-conditioned functional objective derived from a layer reconstruction bound, treating parameter proximity only as a proxy. ConMoE uses unlabeled calibration inputs and keeps the router and expert weights frozen. With the pool fixed, functional assignment improves macro accuracy by 1.82 and 0.35 percentage points at 50% and 75% expert retention, respectively, and also improves pools constructed by other selectors. With task-specific calibration across three MoE architectures and six zero-shot benchmarks, ConMoE achieves the highest macro average among the compared methods at both ratios. At 50% expert retention, ConMoE reduces peak GPU memory by 29.6-48.4% relative to the uncompressed models on our generation workloads. These results establish expert assignment as an independent optimization axis for MoE compression.