mlsanityData to 7 Oct 2026

Expert Merging in Sparse Mixture of Experts with Nash Bargaining

arXiv 2510.16138 ↗First posted 17 Oct 2025cs.LG stat.MLcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Existing expert merging strategies for Sparse Mixture of Experts (SMoE) typically rely on input-dependent or input-independent averaging of expert parameters, but often lack a principled weighting mechanism. In this work, we reinterpret expert merging through the lens of game theory, revealing cooperative and competitive dynamics among experts. Based on this perspective, we introduce Nash Merging of Experts (NAMEx), a novel framework that incorporates Nash Bargaining into the merging process, enabling more balanced and efficient collaboration among experts. Additionally, we incorporate complex momentum into NAMEx to accelerate expert propagation with theoretical guarantees for convergence. Extensive experiments across language modelling, text classification, image classification, and zero-shot robustness under data corruption show that NAMEx consistently outperforms competing methods while integrating seamlessly with popular MoE architectures. Finally, we demonstrate NAMEx's scalability by applying it to large-scale systems, including Qwen1.5-MoE (14B) and DeepSeek-MoE (16B), where it proves effective in both zero-shot and fine-tuning settings. The code is publicly available at: https://github.com/anh147/NAMEx.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 56 of its 85 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

56 of 85 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Tight Clusters Make Specialized Experts21 Feb
  2. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  3. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition4 Feb
  4. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts12 Dec
  5. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy2 Oct
  6. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks7 Jun
  7. 2023Soft Merging of Experts with Adaptive Routing6 Jun
  8. 2022StableMoE: Stable Routing Strategy for Mixture of Experts18 Apr
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
35 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  7. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  8. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  10. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  11. 2023From Sparse to Soft Mixtures of Expertscited by 245
  12. 2021Hash Layers For Large Sparse Modelscited by 236
  13. 2022Multi-Task Learning as a Bargaining Gamecited by 209
  14. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  15. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  16. 2022Patching open-vocabulary models by interpolating weightscited by 187
  17. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  18. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  19. 2018Negative Momentum for Improved Game Dynamicscited by 59
  20. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  21. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  22. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  23. 2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
  24. 2020Scheduled Restart Momentum for Accelerated Stochastic Gradient Descentcited by 24
  25. 2021Fair Exploration via Axiomatic Bargainingcited by 21
  26. 2020MomentumRNN: Integrating Momentum into Recurrent Neural Networkscited by 18
  27. 2017YellowFin and the Art of Momentum Tuningcited by 16
  28. 2023Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?cited by 13
  29. 2023Auxiliary Learning as an Asymmetric Bargaining Gamecited by 12
  30. 2021Complex Momentum for Optimization in Gamescited by 11
  31. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  32. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  33. 2025MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcyclingcited by 5
  34. 2022Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearizationcited by 4
  35. 2025CAMEx: Curvature-aware Merging of Expertscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 9
  2. 2026Pruning and Distilling Mixture-of-Experts into Dense Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
  2. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective
  3. 2024Retraining-Free Merging of Sparse MoE via Hierarchical Clustering
  4. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  5. 2026DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
  6. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  7. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts
  8. 2025CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competition