# Tighter Risk Bounds for Mixtures of Experts

arXiv: https://arxiv.org/abs/2410.10397 · HTML: https://mlsanity.com/paper/2410.10397/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 2 of its 15 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 2 of 2)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [A PAC-Bayesian Tutorial with A Dropout Bound](https://mlsanity.com/paper/1307.2118/) (2013)

## Built on by (most cited first, 3 of 3)

- [Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity](https://mlsanity.com/paper/2410.13964/) (2024)
- [Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs](https://mlsanity.com/paper/2602.15091/) (2026)
- [Expert Routing for Communication-Efficient MoE via Finite Expert Banks](https://mlsanity.com/paper/2605.05278/) (2026)

## Abstract (arXiv)

In this work, we provide upper bounds on the risk of mixtures of experts by imposing local differential privacy (LDP) on their gating mechanism. These theoretical guarantees are tailored to mixtures of experts that utilize the one-out-of-n gating mechanism, as opposed to the conventional n-out-of-n mechanism. The bounds exhibit logarithmic dependence on the number of experts, and encapsulate the dependence on the gating mechanism in the LDP parameter, making them significantly tighter than existing bounds, under reasonable conditions. Experimental results support our theory, demonstrating that our approach enhances the generalization ability of mixtures of experts and validating the feasibility of imposing LDP on the gating mechanism.