mlsanityData to 7 Oct 2026

Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations

arXiv 2601.00457 ↗First posted 1 Jan 2026cs.LG cs.AIcited by 2

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (~0.6) regardless of regularization, and effects on performance are inconsistent -- marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 18 of its 24 references that are in the map. First posted on arXiv 1 Jan 2026.

Builds on

18 of 24 references.

References · 18

  1. 2016Layer Normalizationcited by 5,579
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  9. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  10. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  11. 2017Implicit Regularization in Deep Learningcited by 90
  12. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routingcited by 53
6 more
  1. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Modelscited by 47
  3. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  4. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  5. 2023HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Expertscited by 21
  6. 2025S2MoE: Robust Sparse Mixture of Experts via Stochastic Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Confidence-Adaptive SwiGLU for Mixture-of-Expertscited by 0
  2. 2026A Multi-Modal Generative Model for Tomato Disease Leaves Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  2. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  3. 2026Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization
  4. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Experts
  5. 2026Robustness of Mixtures of Experts to Feature Noise
  6. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  7. 2023Revisiting Single-gated Mixtures of Experts
  8. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective