# Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models

arXiv: https://arxiv.org/abs/2602.14039 · HTML: https://mlsanity.com/paper/2602.14039/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 15 Feb 2026.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 20 of 20)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models](https://mlsanity.com/paper/2406.13233/) (2024)
- [Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models](https://mlsanity.com/paper/2404.05567/) (2024)
- [Advancing Expert Specialization for Better MoE](https://mlsanity.com/paper/2505.22323/) (2025)
- [TWEAC: Transformer with Extendable QA Agent Classifiers](https://mlsanity.com/paper/2104.07081/) (2021)
- [Training Sparse Mixture Of Experts Text Embedding Models](https://mlsanity.com/paper/2502.07972/) (2025)
- [Adversarial Domain Adaptation for Duplicate Question Detection](https://mlsanity.com/paper/1809.02255/) (2018)
- [Ada-K Routing: Boosting the Efficiency of MoE-based LLMs](https://mlsanity.com/paper/2410.10456/) (2024)
- [Maximum Score Routing For Mixture-of-Experts](https://mlsanity.com/paper/2508.12801/) (2025)

## Abstract (arXiv)

Mixture-of-Experts (MoE) embedding models combine expert outputs using weighted linear summation, implicitly assuming a linear subspace structure in the embedding space. This assumption is shown to be inconsistent with the geometry of expert representations. Geometric analysis of a modern MoE embedding model reveals that expert outputs lie on a shared hyperspherical manifold characterized by tightly concentrated norms and substantial angular separation. Under this geometry, linear aggregation induces inward collapse toward the manifold interior, distorting vector magnitude and direction and reducing embedding comparability. To address this inconsistency, Spherical Barycentric Aggregation (SBA) is introduced as a geometry-preserving aggregation operator that separates radial and angular components to maintain hyperspherical structure while remaining fully compatible with existing routing mechanisms. Experiments on selected tasks from the Massive Text Embedding Benchmark (MTEB), including semantic similarity, clustering, and duplicate question detection, demonstrate consistent performance improvements with identical training cost and full stability. Additional geometric analyses confirm that SBA prevents aggregation-induced collapse and preserves hyperspherical consistency, highlighting the importance of geometry-aware aggregation in MoE embedding architectures.