mlsanityData to 7 Oct 2026

Differentially Private Training of Mixture of Experts Models

arXiv 2402.07334 ↗First posted 11 Feb 2024cs.CR cs.LGcited by 2

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing. As Large Language Models (LLMs) scale to billions of parameters, leveraging expansive datasets, they exhibit enhanced linguistic capabilities and emergent abilities. However, this growth raises significant computational and privacy concerns. Our study addresses these issues by exploring the potential of MoE models, known for their computational efficiency, and the application of DP, a standard for privacy preservation. We present the first known attempt to train MoE models under the constraints of DP, addressing the unique challenges posed by their architecture and the complexities of DP integration. Our initial experimental studies demonstrate that MoE models can be effectively trained with DP, achieving performance that is competitive with their non-private counterparts. This initial study aims to provide valuable insights and ignite further research in the domain of privacy-preserving MoE models, softly laying the groundwork for prospective developments in this evolving field.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 25 of its 38 references that are in the map. First posted on arXiv 11 Feb 2024.

Builds on

25 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  11. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
10 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  2. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  3. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  4. 2021Numerical Composition of Differential Privacycited by 174
  5. 2022Reconstructing Training Data with Informed Adversariescited by 143
  6. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  7. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  8. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  9. 2015Efficient Per-Example Gradient Computationscited by 42
  10. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025NoEsis: Differentially Private Knowledge Transfer in Modular LLM Adaptationcited by 1
  2. 2026RAPTOR: Role-Aware Private Training for Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
  2. 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs
  3. 2026MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unification
  4. 2024Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Prompting
  5. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
  6. 2025SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation
  7. 2024SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy
  8. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource