Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
On the map: Mixture-of-experts → MoE theory and gating functions
Abstract (arXiv)
While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ranking, which isolates the evaluation of individual experts and ignores the intricate inter-expert dependencies introduced by the MoE gating network. In this paper, we propose DS-MoE, a theoretically grounded framework that redefines expert selection via difference-of-submodular (DS) optimization. By analyzing the second-order Taylor expansion of the loss degradation, we reveal functional duality within expert combinations: redundancy (where experts encode overlapping representations) and synergy (where experts provide complementary error cancellation). To navigate this duality, we mathematically decouple redundancy reduction from synergy maximization by formulating the selection objective as a DS function. Furthermore, we devise a tailored majorization-minimization (MM) algorithm with provable monotonicity guarantees to efficiently identify the optimal expert subset. Extensive experiments demonstrate that DS-MoE effectively preserves indispensable expert combinations, achieving superior performance compared to the state-of-the-art baselines.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 38 of its 50 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
38 of 50 references.
Closest ancestors · 1
Same direction, newest first
Other references · 37
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2024Mixtral of Expertscited by 1,785
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2022Mixture-of-Experts with Expert Choice Routingcited by 611
25 more
- 2014New insights and perspectives on the natural gradient methodcited by 435
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
- 2022Unified Scaling Laws for Routed Language Modelscited by 249
- 2023From Sparse to Soft Mixtures of Expertscited by 245
- 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
- 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
- 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
- 2021FastMoE: A Fast Mixture-of-Expert Training Systemcited by 110
- 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
- 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
- 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
- 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 46
- 2023FedSN: A Federated Learning Framework over Heterogeneous LEO Satellite Networkscited by 38
- 2023Optimal Resource Allocation for U-Shaped Parallel Split Learningcited by 33
- 2024Scaling Law for Time Series Forecastingcited by 31
- 2013Stochastic Majorization-Minimization Algorithms for Large-Scale Optimizationcited by 28
- 2025DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruningcited by 24
- 2024SatFed: A Resource-Efficient LEO Satellite-Assisted Heterogeneous Federated Learning Frameworkcited by 21
- 2014Algorithms for Approximate Minimization of the Difference Between Submodular Functions, with Applicationscited by 16
- 2024IFViT: Interpretable Fixed-Length Representation for Fingerprint Matching via Vision Transformercited by 16
- 2024Scattered Mixture-of-Experts Implementationcited by 15
- 2025HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systemscited by 15
- 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
- 2025Optimizing Split Federated Learning with Unstable Client Participationcited by 5
- 2026NSC-SL: A Bandwidth-Aware Neural Subspace Compression for Communication-Efficient Split Learningcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
- 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
- 2025Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
- 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
- 2025GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
- 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
- 2026Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
- 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts