# Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs

arXiv: https://arxiv.org/abs/2602.15091 · HTML: https://mlsanity.com/paper/2602.15091/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 16 Feb 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 7 of 7)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [Hierarchically Structured Task-Agnostic Continual Learning](https://mlsanity.com/paper/2211.07725/) (2022)
- [An Information-theoretic On-line Learning Principle for Specialization in Hierarchical Decision-Making Systems](https://mlsanity.com/paper/1907.11452/) (2019)
- [Tighter Risk Bounds for Mixtures of Experts](https://mlsanity.com/paper/2410.10397/) (2024)

## Built on by (most cited first, 2 of 2)

- [Expert Routing for Communication-Efficient MoE via Finite Expert Banks](https://mlsanity.com/paper/2605.05278/) (2026)
- [Sparse In-Network Learning via Shortest-Path Backpropagation and Finite-Rate Gating](https://mlsanity.com/paper/2605.23424/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures decompose prediction tasks into specialized expert sub-networks selected by a gating mechanism. This letter adopts a communication-theoretic view of MoE gating, modeling the gate as a stochastic channel operating under a finite information rate. Within an information-theoretic learning framework, we specialize a mutual-information generalization bound and develop a rate-distortion characterization D(Rg) of finite-rate gating, where Rg:=I(X; T), yielding (under a standard empirical rate-distortion optimality condition) mathbbE[R(W)] ≤ D(Rg)+δm+√(2/m), I(S; W). The analysis yields capacity-aware limits for communication-constrained MoE systems, and numerical simulations on synthetic multi-expert models empirically confirm the predicted trade-offs between gating rate, expressivity, and generalization.