# Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds

arXiv: https://arxiv.org/abs/2602.17798 · HTML: https://mlsanity.com/paper/2602.17798/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 16 of 16)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Scaling Laws for Fine-Grained Mixture of Experts](https://mlsanity.com/paper/2402.07871/) (2024)
- [StableMoE: Stable Routing Strategy for Mixture of Experts](https://mlsanity.com/paper/2204.08396/) (2022)
- [Soft Merging of Experts with Adaptive Routing](https://mlsanity.com/paper/2306.03745/) (2023)
- [Harder Tasks Need More Experts: Dynamic Routing in MoE Models](https://mlsanity.com/paper/2403.07652/) (2024)
- [Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization](https://mlsanity.com/paper/2509.03733/) (2025)
- [Spectral Manifold Regularization for Stable and Modular Routing in Deep MoE Architectures](https://mlsanity.com/paper/2601.03889/) (2026)
- [Beyond Variance: Knowledge-Aware LLM Compression via Fisher-Aligned Subspace Diagnostics](https://mlsanity.com/paper/2601.07197/) (2026)
- [Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction](https://mlsanity.com/paper/2602.12204/) (2026)

## Built on by (most cited first, 2 of 2)

- [Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.12476/) (2026)
- [Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models](https://mlsanity.com/paper/2609.37751/) (2026)

## Abstract (arXiv)

Mixture-of-Experts models rely on learned routers to assign tokens to experts, yet standard softmax gating provides no principled mechanism to control the tradeoff between sparsity and utilization. We propose Grassmannian MoE (GrMoE), a routing framework that operates on the Grassmannian manifold of subspaces, where gating weights arise from the concentration parameters of Matrix Bingham distributions. This construction yields a single, interpretable knob -- the concentration matrix Lambda -- that continuously controls routing entropy, replacing discrete top-k selection with a smooth, geometrically principled sparsity mechanism. We further develop an amortized variational inference procedure for posterior routing distributions, enabling uncertainty-aware expert assignment that naturally resists expert collapse. We formally prove tight bounds relating the Bingham concentration spectrum to routing entropy, expected top-k mass, and an exponential bound on expert collapse, establishing the first formal theory of concentration-controlled sparsity. On synthetic routing tasks, a 350M-parameter MoE language model with 8 experts, a 1.3B-parameter model with 16 experts, and a 2.7B-parameter model with 32 experts, GrMoE achieves 0% routing collapse across all seeds, comparable or better perplexity with 15--30% improved load balance, and a smooth monotonic relationship between concentration and effective sparsity that enables post-hoc sparsity tuning without retraining. Token-level analysis reveals that experts learn heterogeneous concentration values that correlate with linguistic specialization, providing interpretable routing behavior.