Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
Mixture-of-Experts (MoE) architectures achieve scalable capacity through sparse routing, yet the geometric structure of expert specialization remains poorly understood. We introduce a unified Jacobian-PCA-Grassmann framework for analyzing MoE layers in both function space and representation space. Across pretrained MoE Transformers (Mistral, Qwen), we find a consistent structural asymmetry: experts exhibit strong functional decorrelation (consistently low, near-zero cross-expert Jacobian alignment) while their routed representations occupy distinct but partially overlapping subspaces. This indicates that functional decorrelation and representation overlap coexist rather than coincide in MoE specialization. Controlled routing experiments further indicate that routing sparsity appears to be a key factor shaping this geometry: top-k routing induces sharper functional separation and larger subspace divergence, whereas fully soft routing yields more entangled expert structure. Together, these results suggest a geometric interpretation in which MoE layers may be viewed as implementing locally decorrelated operators over overlapping submanifolds on a shared representation manifold, and provide a general diagnostic framework for studying conditional computation in modern Transformer architectures.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 17 of its 20 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
17 of 20 references.
Closest ancestors · 2
Same direction, newest first
- 2025Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning3 Sep
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 15
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
- 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
- 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
- 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
- 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 156
- 2018The Emergence of Spectral Universality in Deep Networkscited by 74
- 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
- 2024Mixture Compressor for Mixture-of-Experts LLMs Gains Morecited by 41
- 2025Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Mergingcited by 28
3 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization
- 2025Optimal Scaling Laws for Efficiency Gains in a Theoretical Transformer-Augmented Sectional MoE Framework
- 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
- 2025Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer
- 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers
- 2026Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
- 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
- 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise