mlsanityData to 7 Oct 2026

Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression

arXiv 2510.02345 ↗First posted 27 Sep 2025cs.CL cs.AI cs.DCcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead. We introduce a unified framework based on dynamic expert clustering and structured compression to address these issues cohesively. Our method employs an online clustering procedure that periodically regroups experts using a fused metric of parameter and activation similarity, which stabilizes expert utilization. To our knowledge, this is one of the first frameworks to leverage the semantic embedding capability of the router to dynamically reconfigure the model's architecture during training for substantial efficiency gains. Within each cluster, we decompose expert weights into a shared base matrix and extremely low-rank residual adapters, achieving up to fivefold parameter reduction per group while preserving specialization. This structure enables a two-stage hierarchical routing strategy: tokens are first assigned to a cluster, then to specific experts within it, drastically reducing the routing search space and the volume of all-to-all communication. Furthermore, a heterogeneous precision scheme, which stores shared bases in FP16 and residual factors in INT4, coupled with dynamic offloading of inactive clusters, reduces peak memory consumption to levels comparable to dense models. Evaluated on GLUE and WikiText-103, our framework matches the quality of standard MoE models while reducing total parameters by approximately 80%, improving throughput by 10% to 20%, and lowering expert load variance by a factor of over three. Our work demonstrates that structural reorganization is a principled path toward scalable, efficient, and memory-effective MoE LLMs. Code is available at https://github.com/szdtzpj/Breaking_the_moe_trilemma

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 24 of its 29 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

24 of 29 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging29 Jun
  2. 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free14 Oct
  3. 2024OLMoE: Open Mixture-of-Experts Language Models3 Sep
  4. 2024Mixtral of Experts8 Jan
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 17

  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  8. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  9. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  10. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  11. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 71
  12. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
5 more
  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 53
  2. 2025Mixture of Lookup Expertscited by 11
  3. 2024PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Modelcited by 9
  4. 2025A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation Modelscited by 9
  5. 2022SMILE: Scaling Mixture-of-Experts with Efficient Bi-level Routingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoringcited by 21
  2. 2026Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinementcited by 13
  3. 2026OrderMoE: An expert similarity driven distributed edge MoE inferencecited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Cluster-Driven Expert Pruning for Mixture-of-Experts Large Language Models
  2. 2026Effective MoE-based LLM Compression by Exploiting Heterogeneous Inter-Group Experts Routing Frequency and Information Density
  3. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
  4. 2022MoEC: Mixture of Expert Clusters
  5. 2025Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
  6. 2025Advancing Expert Specialization for Better MoE
  7. 2024Mixture Compressor for Mixture-of-Experts LLMs Gains More
  8. 2026Shape Mutating Expert Compression:LorExperts and BTExperts