# ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization

arXiv: https://arxiv.org/abs/2511.10971 · HTML: https://mlsanity.com/paper/2511.10971/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 14 Nov 2025.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts](https://mlsanity.com/paper/2408.15664/) (2024)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [On the Representation Collapse of Sparse Mixture of Experts](https://mlsanity.com/paper/2204.09179/) (2022)
- [StableMoE: Stable Routing Strategy for Mixture of Experts](https://mlsanity.com/paper/2204.08396/) (2022)
- [Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?](https://mlsanity.com/paper/1810.09102/) (2018)
- [Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models](https://mlsanity.com/paper/2501.11873/) (2025)
- [Advancing Expert Specialization for Better MoE](https://mlsanity.com/paper/2505.22323/) (2025)
- [Global-Local Transformer for Brain Age Estimation](https://mlsanity.com/paper/2109.01663/) (2021)
- [EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing](https://mlsanity.com/paper/2410.02098/) (2024)
- [Routers in Vision Mixture of Experts: An Empirical Study](https://mlsanity.com/paper/2401.15969/) (2024)
- [CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling](https://mlsanity.com/paper/2409.19291/) (2024)
- [Improving Expert Specialization in Mixture of Experts](https://mlsanity.com/paper/2302.14703/) (2023)
- [R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts](https://mlsanity.com/paper/2502.20395/) (2025)
- [Revisiting Single-gated Mixtures of Experts](https://mlsanity.com/paper/2304.05497/) (2023)
- [FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Text](https://mlsanity.com/paper/2507.10095/) (2025)
- [Triamese-ViT: A 3D-Aware Method for Robust Brain Age Estimation from MRIs](https://mlsanity.com/paper/2401.09475/) (2024)

## Built on by (most cited first, 7 of 7)

- [φ-Balancing for Mixture-of-Experts Training](https://mlsanity.com/paper/2605.15403/) (2026)
- [Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2605.12476/) (2026)
- [Beyond Conditional Computation: Retrieval-Augmented Genomic Foundation Models with Gengram](https://mlsanity.com/paper/2601.22203/) (2026)
- [CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering](https://mlsanity.com/paper/2604.16930/) (2026)
- [Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs](https://mlsanity.com/paper/2606.09038/) (2026)
- [APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference](https://mlsanity.com/paper/2608.11688/) (2026)
- [Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models](https://mlsanity.com/paper/2609.39445/) (2026)

## Abstract (arXiv)

Mixture-of-Experts (MoE) architectures expand model capacity by sparsely activating experts but face two core challenges: misalignment between router logits and each expert's internal structure leads to unstable routing and expert underutilization, and load imbalances create straggler bottlenecks. Standard solutions, such as auxiliary load-balancing losses, can reduce load disparities but often weaken expert specialization and hurt downstream performance. To address these issues, we propose ERMoE, a sparse MoE transformer that reparameterizes each expert in a learned orthonormal eigenbasis and replaces learned gating logits with an "Eigenbasis Score", defined as the cosine similarity between input features and an expert's basis. This content-aware routing ties token assignments directly to experts' representation spaces, stabilizing utilization and promoting interpretable specialization without sacrificing sparsity. Crucially, ERMoE removes the need for explicit balancing losses and avoids the interfering gradients they introduce. We show that ERMoE achieves state-of-the-art accuracy on ImageNet classification and cross-modal image-text retrieval benchmarks (e.g., COCO, Flickr30K), while naturally producing flatter expert load distributions. Moreover, a 3D MRI variant (ERMoE-ba) improves brain age prediction accuracy by more than 7% and yields anatomically interpretable expert specializations. ERMoE thus introduces a new architectural principle for sparse expert models that directly addresses routing instabilities and enables improved performance with scalable, interpretable specialization.