mlsanityData to 14 Oct 2026

Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts

arXiv 2610.15039 ↗First posted 10 Oct 2026cs.LG stat.MLcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture-of-Experts (MoE) models combine specialized experts through an input-dependent routing mechanism, and their sparse variants underpin many modern large-scale systems. Yet the statistical cost of adaptive routing remains only partially understood. We study the estimation error of least-squares MoE estimators when both the experts and the gating network are Lipschitz in their parameters. We represent each MoE as a gate--expert pair in a Gated Hilbert Space, whose norm |·|_Delta separates the contributions of routing and expert discrepancies. Using its empirical representation, we derive an oracle inequality with a fast rate of order tildemathcal Obig((Kdpr,e+dpr,g)/nbig) for dense MoEs with K experts. For sparse Top-k and Top-p routing, the selection map is discontinuous in the gating parameters. We handle this by partitioning the gating class according to the selection patterns it induces on the sample. The resulting bounds match the dense rate up to an additive routing cost governed by the Natarajan dimension of the selector. We show that such a routing-complexity condition cannot be dropped: Lipschitz continuity in the parameters alone does not suffice. For linear softmax gating, the routing cost is of the same order as the gating term. Finally, for linear experts, we establish a matching minimax lower bound of order Kdpr,e/n, showing that the rates are optimal up to logarithmic factors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 10 of its 48 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

10 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Convergence Rates for Softmax Gating Mixture of Experts5 Mar
  2. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts22 May
  3. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study26 Mar
  4. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts25 Sep

Other references · 6

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  5. 2015Learning with Square Loss: Localization through Offset Rademacher Complexitycited by 38
  6. 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards a Statistical Understanding of Mixture-of-Experts
  2. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  3. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Compute
  4. 2026Expert Routing for Communication-Efficient MoE via Finite Expert Banks
  5. 2026Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
  6. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  7. 2026How Sparse Probability Maps Shape Mixture-of-Experts Routing
  8. 2026Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts