mlsanityData to 7 Oct 2026

MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification

arXiv 2605.14289 ↗First posted 14 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models scale capacity by combining specialized experts, but most existing approaches assume centralized access to training data. In practice, data are distributed across clients and cannot be shared due to privacy constraints, making unified MoE training challenging. We propose MetaMoE, a privacy-preserving framework that unifies independently trained, domain-specialized experts into a single MoE using public proxy data as surrogates for inaccessible private data. Central to MetaMoE is diversity-aware proxy selection, which selects client-domain-relevant and diverse samples from public data to effectively approximate private data distributions and supervise router learning. These proxies are further used to align expert training, improving expert coordination at unification time, while a context-aware router enhances expert selection across heterogeneous inputs. Experiments on computer vision and natural language processing benchmarks demonstrate that MetaMoE consistently outperforms recent privacy-preserving MoE unification methods. Code is available at https://github.com/ws-jiang/MetaMoE.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 32 of its 59 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

32 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  10. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  11. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
19 more
  1. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  2. 2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 307
  3. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  4. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  5. 2021Hash Layers For Large Sparse Modelscited by 236
  6. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  7. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  8. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  9. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  10. 2022Trading Off Privacy, Utility and Efficiency in Federated Learningcited by 33
  11. 2024GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoningcited by 31
  12. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  13. 2017Faster Greedy MAP Inference for Determinantal Point Processescited by 14
  14. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  15. 2023Exploring adversarial attacks in federated learning for medical imagingcited by 8
  16. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  17. 2024On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialistscited by 6
  18. 2025LoRASuite: Efficient LoRA Adaptation Across Large Language Model Upgradescited by 4
  19. 2024Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategycited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RAPTOR: Role-Aware Private Training for Mixture-of-Experts
  2. 2024Differentially Private Training of Mixture of Experts Models
  3. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
  4. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs
  5. 2026SecMoE: Communication-Efficient Secure MoE Inference via Select-Then-Compute
  6. 2025MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
  7. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  8. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts