mlsanityData to 14 Oct 2026

Retraining-Free Merging of Sparse MoE via Hierarchical Clustering

arXiv 2410.08589 ↗First posted 11 Oct 2024cs.LGcited by 35

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Sparse Mixture-of-Experts (SMoE) models represent a significant advancement in large language model (LLM) development through their efficient parameter utilization. These models achieve substantial performance improvements at reduced inference costs. However, the deployment of SMoE models faces constraints from extensive memory requirements of expert components in resource-limited environments. To address these limitations, this paper introduces Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE), a task-agnostic expert merging framework for parameter reduction without retraining. HC-SMoE introduces a novel hierarchical clustering approach based on expert outputs to ensure merging robustness independent of routing decisions. The proposed output-based clustering method enables effective capture of functional relationships between experts for large-scale architectures. We provide theoretical analysis and comprehensive evaluations across multiple zero-shot language tasks to demonstrate HC-SMoE's effectiveness in state-of-the-art models including Qwen and Mixtral. The experimental results validate HC-SMoE's superior performance and practical applicability for real-world deployments.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 21 of its 33 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

21 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models22 Feb
  2. 2024Mixtral of Experts8 Jan
  3. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer15 Oct
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  10. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  11. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  12. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
5 more
  1. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  2. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
  3. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Expertscited by 72
  4. 2024Training-Free Pretrained Model Mergingcited by 32
  5. 2021CodeQA: A Question Answering Dataset for Source Code Comprehensioncited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 29.

35 papers.

Sort:

By area:

  1. 2025MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMscited by 9
  2. 2025PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inferencecited by 9
  3. 2026EvoESAP: Non-Uniform Expert Pruning for Sparse MoEcited by 9
  4. 2025CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysiscited by 7
  5. 2025MoE Pathfinder: Trajectory-driven Expert Pruningcited by 7
  6. 2026How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principlecited by 6
  7. 2026AIMER: Calibration-Free Task-Agnostic MoE Expert Pruningcited by 5
  8. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compressioncited by 4
  9. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
  10. 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measurescited by 3
  11. 2025HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Spacecited by 3
  12. 2026SERE: Similarity-based Expert Re-routing for Efficient Batch Decoding in MoE Modelscited by 3
23 more
  1. 2026LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacingcited by 2
  2. 2026Less is MoE: Trimming Experts in Domain-Specialist Language Modelscited by 2
  3. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  4. 2026UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Modelscited by 2
  5. 2026HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Expertscited by 1
  6. 2026BEAM: Binary Expert Activation Masking for Dynamic Routing in MoEcited by 1
  7. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1
  8. 2026ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compressioncited by 1
  9. 2026On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domaincited by 1
  10. 2026Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Modelscited by 1
  11. 2026Shape Mutating Expert Compression:LorExperts and BTExpertscited by 0
  12. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEscited by 0
  13. 2026Residual Sparsification via Output Importance for Compressing Mixture-of-Experts LLMscited by 0
  14. 2026When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Modelscited by 0
  15. 2026Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustmentcited by 0
  16. 2026RAZOR: Pruning Replaceable Experts in LLMscited by 0
  17. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuitcited by 0
  18. 2026SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Servingcited by 0
  19. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximationcited by 0
  20. 2026Dynamic Expert Pruning for Multi-Agent Systemscited by 0
  21. 2026SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretrainingcited by 0
  22. 2026PRIME: Frequency-Aware Unified Pruning and Merging for MoE Compressioncited by 0
  23. 2026SLEM: Spectrum-Aware Latent Expert Mergingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022MoEC: Mixture of Expert Clusters
  2. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  3. 2025Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression
  4. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  5. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  6. 2025Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
  7. 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling
  8. 2024Multi-Head Mixture-of-Experts