mlsanityData to 7 Oct 2026

MergeMoE: Efficient Compression of MoE Models via Expert Output Merging

arXiv 2510.14436 ↗First posted 16 Oct 2025cs.LGcited by 8

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However, the substantial memory overhead of MoE models has made their compression an important research direction. In this work, we provide a theoretical analysis of expert merging, a recently proposed technique for compressing MoE models. Rather than interpreting expert merging from the conventional perspective of parameter aggregation, we approach it from the perspective of merging experts' outputs. Our key insight is that the merging process can be interpreted as inserting additional matrices into the forward computation, which naturally leads to an optimization formulation. Building on this analysis, we introduce MergeMoE, a method that leverages mathematical optimization to construct the compression matrices. We evaluate MergeMoE on multiple MoE models and show that our algorithm consistently outperforms the baselines with the same compression ratios.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 31 of its 38 references that are in the map. First posted on arXiv 16 Oct 2025.

Builds on

31 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router15 Oct
  3. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models3 Jun
  4. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollars11 Apr
  5. 2024Mixtral of Experts8 Jan
  6. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Experts15 Oct
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2020Knowledge Distillation: A Surveycited by 1,578
12 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  3. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  6. 2023DepGraph: Towards Any Structural Pruningcited by 202
  7. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  8. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  9. 2022Fusing finetuned models for better pretrainingcited by 110
  10. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  11. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  12. 2024Towards Meta-Pruning via Optimal Transportcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  2. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Densitycited by 6
  3. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  4. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1
  5. 2026Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Modelscited by 1
  6. 2026Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspectivecited by 0
  7. 2026Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlapcited by 0
  8. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
  2. 2025MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs
  3. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
  4. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  5. 2026Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
  6. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
  7. 2025MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?
  8. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing