mlsanityData to 14 Oct 2026

PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs

arXiv 2506.02965 ↗First posted 3 Jun 2025cs.LGcited by 1

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) has been gaining popularity due to its successful adaptation to large language models (LLMs). In this work, we introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE), which leverages the sparsity of the MoE architecture for memory-efficient decentralized collaborative LLM training, enabling multiple parties with limited GPU-memory and data resources to collectively train more capable LLMs than they could achieve individually. At the same time, this approach protects training data privacy of each participant by keeping training data, as well as parts of the forward pass signal and gradients locally within each party. By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances. Unlike most privacy-preserving schemes, which pay for confidentiality with lower task accuracy, our framework breaks that trade-off: across seven popular LLM benchmarks, it almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 55 of its 61 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

55 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Mixture of A Million Experts4 Jul
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  10. 2022Emergent Abilities of Large Language Modelscited by 2,512
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  5. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  6. 2019Deep Leakage from Gradientscited by 1,021
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2019Towards Federated Learning at Scale: System Designcited by 832
  9. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  10. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  13. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  14. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  15. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  16. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  17. 2024Are We Done with MMLU?cited by 194
  18. 2020The Computational Limits of Deep Learningcited by 193
  19. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  20. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  21. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  22. 2022MPCFormer: fast, performant and private Transformer inference with MPCcited by 69
  23. 2022THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryptioncited by 57
  24. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  25. 2021CryptGPU: Fast Privacy-Preserving Machine Learning on the GPUcited by 46
  26. 2022Do Gradient Inversion Attacks Make Federated Learning Unsafe?cited by 44
  27. 2021Gradient-Leakage Resilient Federated Learningcited by 40
  28. 2022A Survey on Gradient Inversion: Attacks, Defenses and Future Directionscited by 40
  29. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 39
  30. 2024Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inferencecited by 28
  31. 2025FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Modelscited by 17
  32. 2024DAGER: Exact Gradient Inversion for Large Language Modelscited by 14
  33. 2024Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architecturescited by 12
  34. 2024TransLinkGuard: Safeguarding Transformer Models Against Model Stealing in Edge Deploymentcited by 11
  35. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
  36. 2024Improving Robustness to Model Inversion Attacks via Sparse Coding Architecturescited by 8
  37. 2024Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedulescited by 7
  38. 2025Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routingcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Does Every User Need a Private LoRA? Decoupling Personalization from Per-User Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Differentially Private Training of Mixture of Experts Models
  2. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs
  3. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  4. 2026RAPTOR: Role-Aware Private Training for Mixture-of-Experts
  5. 2025MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs
  6. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
  7. 2026MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
  8. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm