Tighter Risk Bounds for Mixtures of Experts
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
In this work, we provide upper bounds on the risk of mixtures of experts by imposing local differential privacy (LDP) on their gating mechanism. These theoretical guarantees are tailored to mixtures of experts that utilize the one-out-of-n gating mechanism, as opposed to the conventional n-out-of-n mechanism. The bounds exhibit logarithmic dependence on the number of experts, and encapsulate the dependence on the gating mechanism in the LDP parameter, making them significantly tighter than existing bounds, under reasonable conditions. Experimental results support our theory, demonstrating that our approach enhances the generalization ability of mixtures of experts and validating the feasibility of imposing LDP on the gating mechanism.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 2 of its 15 references that are in the map. First posted on arXiv 14 Oct 2024.
Builds on
2 of 15 references.
References · 2
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2013A PAC-Bayesian Tutorial with A Dropout Boundcited by 43
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 2.
3 papers.
- 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
- 2026Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offscited by 2
- 2026Expert Routing for Communication-Efficient MoE via Finite Expert Bankscited by 1
- 2026Expert Routing for Communication-Efficient MoE via Finite Expert Banks6 May 2026
- 2026Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs16 Feb 2026
- 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity17 Oct 2024
Similar, not cited
Close in wording, no citation either way
- 2026RAPTOR: Role-Aware Private Training for Mixture-of-Experts
- 2025PWC-MoE: Privacy-Aware Wireless Collaborative Mixture of Experts
- 2024Differentially Private Training of Mixture of Experts Models
- 2025KL-regularization Itself is Differentially Private in Bandits and RLHF
- 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
- 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs
- 2026Shuffle and Joint Differential Privacy for Generalized Linear Contextual Bandits
- 2026Privacy Without Regret: Differentially Private Inference-Time Alignment